Closing the communications gap between data engineers and data consumers

Businesses that effectively leverage data consistently outperform their competition. That’s why modern business leaders are leaning into data-driven decision making like never before. But according to research by NewVantage Partners, while 99% of executives aspire to build data-driven cultures, only 32% believe they’ve actually done so. What’s the problem?

A significant challenge is the communications gap between producers of data, like data engineers, and consumers of data, like analysts and business users. Data producers are inundated with ad hoc questions about the quality and status of data and reports – as much as 30% of their time. Yet, data consumers are often in the dark about the origin, status, and quality of the data they are using, leading to uncertainty or even bypassing the data altogether. 

Our new DataOps application helps your team close this communications gap. The application automates notifications on important information about data, like data quality and usage, and creates a seamless way for data producers and consumers to communicate within the BI, productivity, and data integration tools they use every day.

The communication gap

Data-driven decision making is a team sport. It works best when there’s continuous communication between data producers and data users. 

The problem is, this kind of feedback loop is difficult to build. Without guidance, data consumers often feel overwhelmed by the volume of data available to them and consequently struggle to understand if they can rely on it to inform important decisions. An unfortunate effect of this confusion can be the rise of "shadow BI" – people promoting alternate, incorrect data in an effort to simply get the job done.

On the other side of the equation, proactive data consumers will resort to email or messaging tools to ask questions about data and reports. This barrage of disparate messages takes data engineers away from important data work instead they are answering, and re-answering, the same questions or providing status updates when their time is better served fixing issues. 

The result? Finger-pointing and "data-driven procrastination." Data consumers blame data teams for a lack of guidance or a lack of progress on data issues and often resort to making decisions without data. Data engineers are stuck between a rock and a hard place: either address the data issues and leave consumers in the dark, or respond to data consumers and spend less time actually ensuring the reliability of the benefit of data. 

BB Bot makes communication easy

The DataOps application uses automations to surface contextual insights and real-time data quality updates from tools like dbt, Airflow, and Matillion to data consumers directly within BI, productivity, and communications tools like Looker Studio, Tableau, ThoughtSpot, Slack, Salesforce and more. 

Inspired by the Owl of Athena — the Goddess of wisdom in Greek mythology — “Bubo,” or “BB Bot” for short, provides a seamless way for data teams to update data consumers on the health of their data, and communicate important information and receive feedback in real time. This increases confidence in — and use of — high quality data. BB Bot serves as your data teams' stamp of approval on a data report, signaling to users that the information has been verified and can be trusted.

These AI monitoring agents send automated notifications — “Hoots” — in real-time, surfacing critical communications directly within applications and automatically flagging all affected data and reports with crucial information. The result is the seamless communication of important DataOps knowledge without the burden of sending disparate messages and answering, and re-answering, ad hoc questions.

The Hoots automatically surface a wealth of important information, including:

• Metric definitions

• Data lineage on data sets and reports

• Data quality updates, including pipeline health and freshness

• Report lifecycle status 

• Analytics on the reports, including usage and ownership

• Business glossary definitions 

• Related policy and PII information

A data catalog is the perfect place for DataOps 

data.world is recognized by Forrester as a leading enterprise data catalog for DataOps, with robust data lineage, automated workflows, and a cloud native architecture that complements the modern data stack. The AI-powered BB Bots and automated Hoots form the core of our data catalog’s new DataOps Applications.

BB Bot will be available free of charge to any data.world user subscribed at our standard tier and above. Want to learn more about Bubo and Hoots? Schedule a customized demo with a data.world expert now.