data.world has officially leveled up its integration with Snowflake’s new data quality capabilities
data.world enables trusted conversations with your company’s data and knowledge with the AI Context Engine™
Accelerate adoption of AI with the AI Context Engine™️, now generally available
Understand the broad spectrum of search and how knowledge graphs are enabling data catalog users to explore far beyond data and metadata.
Join our Demo Day to see how businesses are transforming the way they think about and use data with a guided tour through the extraordinary capabilities of data.world's data catalog platform.
Are you ready to revolutionize your data strategy and unlock the full potential of AI in your organization?
Come join us in our mission to deliver data for all and data for good!
Are you ready to revolutionize your data strategy and unlock the full potential of AI in your organization?
Data governance is about understanding and managing the data within an organization. Traditionally, companies use data governance as a risk avoidance tactic or a way to make sure their data team is being compliant with strict policies around data governance.
People often confuse data governance with data management. This chart explains the difference.
The practice of data governance continues to evolve as organizations recognize the transformative power of data and metadata. Like we said before, the traditional top-down approach to data governance often focuses on risk and compliance. However, modern, data-driven organizations take a more holistic approach to data governance where everyone participates in the practice and focuses on learning and improving data over time. This is referred to as Agile Data Governance.
Before we get into agile, let’s first explore data governance using a top-down approach.
For decades, companies have been using a top-down approach to manage their data. This method allows top-level executives to create strict policies and define best practices for their data team. Meanwhile, data consumers are anxiously waiting to get their jobs done, but they aren’t able to until they receive instructions.
This is only one downfall of traditional data governance. In the book, Winning with Data, Tomasz Tunguz describes five main challenges companies must overcome to create data-driven cultures. These problems stem from issues in both poor data governance and data management.
Data consumers make request after request to data producers for the same data, yet the data stewardship committees only prioritize the loudest or most urgent data needs. Data consumers become frustrated with the delay in getting what they need.
Everyone has encountered that “one person” who has built a one-off system to produce some analytics or scripts that only run on his or her laptop. Prominent top-down governance practices encourage people to create rogue databases and processes since they don’t want to deal with the imposed bureaucracy.
In most organizations, those who try to understand the availability and use cases of data assets encounter inefficiencies, partial answers, and confusing systems. Top down, non iterative governance practices make this even more of a chore and discourages people from participating. You’re better off if you capture this information as work is getting done.
When data work isn’t transparent, people don’t trust it. They show up with different versions of the same analysis after months of work. They argue about small details, data sources, even project goals. This slows down your analytics projects each time they happen.
You can’t have a data-driven culture if your people don’t understand the basic workings of statistics. They need to appreciate, and have simple ways to follow, the scientific method and other best practices that make analytics valid and useful.
As data evolves, we continually adopt new processes and methodologies. In the developmental stages of most products, there are two methodologies to follow; waterfall or agile. Earlier on, many adopted a waterfall approach to data governance, which is highly sequential and linear. As we see a shift in data and technology development, many are taking the agile approach.
It’s time we shift the definition of data governance to a process of doing data and analytics work together. Create an inclusive, collaborative data culture by adopting agile data governance.
Agile Data Governance is the process of creating and improving data assets by iteratively capturing knowledge as data producers and consumers work together so that everyone can benefit. It adapts the deeply proven best practices of Agile and Open software development to data and analytics.
The goal of Agile is to tighten the cycle time between data producers and data consumers. Instead of having a top-down relationship with your team, move towards enabling them to work together in an iterative cycle so they can make that cycle faster and more efficient. This is how to unlock the value of your data supply chain and move towards building a data-driven culture.
Building a data-driven culture with a top-down approach where every detail is planned far in advance leaves data consumers and data producers blocked until they get their marching orders. But there’s wisdom in the saying, “How do you eat an elephant? One bite at a time.” Agile Data Governance gives us a way to build an efficient data supply chain and create a data-driven culture one bite at a time.
Let’s revisit the key issues of top-down methodologies. Comparing traditional data governance to an agile approach will show you exactly how Agile Data Governance can free your data supply chain from blockers so you can start taking small bites to move toward a data-driven culture.
By using agile principles, data producers, data consumers, and domain experts iterate together to build reusable assets that lower the frequency of ad-hoc requests.
Data consumers have a direct, clear way to request and iterate on data assets. This reduces the prevalence of “emailed spreadsheets.” Plus, data assets will be well-documented in a data catalog, so more people can find, understand, and use them.
In Agile Data Governance, you do the documentation while you do the work. This near-real-time documentation decreases your team’s knowledge debt and gives them clarity on what data exists, what it means, and how to use it.
With Agile Data Governance, transparency means course correction and peer review happens as analysis unfolds. This creates a shared understanding which can be poured into business glossaries and other alignment tools.
An agile process encourages participation with, and observation of, talented people doing amazing work. This increases data literacy and skill across your entire company.
Like any culture change, the people in your company need to become your priority. Think about the team who will execute this change and implement these processes. Most companies encompass all of these roles on their data team:
Looking at the organization breakdown above, think about the structure of your data team from both a top-down and bottom-up perspective. There needs to be a strategic team who will be extracting the best practices for your team to use. At the same time, keep in mind that you also have data producers and consumers actively working on analytics projects.
Bridging the gap between your strategic team and data consumers is a critical step in creating an inclusive, data-driven organization. It eliminates the silos that were built through previous top-down efforts. After closing that gap, data consumers, data producers, and domain experts can easily collaborate to find the answers to critical business questions. Now that we have a clear understanding of our data team breakdown and how they need to be working with one another, start thinking about the processes that they need to carry out to drive more value with data.
Once again, we want to take the deeply proven best practices of Agile and Open software development and apply them to our Agile Data Governance process.
You can see how data producers quickly learn what’s working and what’s not about the data sources they’re curating so they can make improvements in real-time. Doing analytics with a living, evolving data asset focuses stakeholders and provides valuable insights at high frequency. That’s why Agile Data Governance practitioners see ROI in days instead of month.
By cataloging the work as it happens in your data catalog, and not only the “finished” analysis, teams continuously learn from each other and elevate their data literacy. That’s because people learn data skills and domain knowledge faster by doing the work and seeing their peers solve real problems.
Getting into a motion of constant iteration and knowledge capture will help your organization not only get faster answers to current issues, but ensure your data is reusable for future projects. To get here, make sure you have the right foundation and tools in place to set you up for success.
Only some tools are right for Agile Data Governance, in the same way the growth of Agile and Open-source software development demanded new tools. Agile software development meant throwing out heavyweight requirements docs and architecture diagrams that would take weeks to write.
While there are great tools that will help you implement agile on a data level, they will not completely support the processes that make Agile Data Governance valuable.
Is there a way to support this agility that companies are looking for?
Yes; the answer is DataOps.
DataOps is a management methodology that considers the people, processes, and tools involved in making your data supply chain more efficient. It encompasses an end-to-end approach for data management that produces a unified view of the entire data lifecycle.
Putting a collaborative data catalog at the center of your DataOps and Agile Data Governance processes is essential to getting a clear picture of your entire data supply chain.
Getting the processes down is one thing, but making sure you are constantly improving them is another. Data-driven cultures use what they have learned and apply that to their processes to make them more efficient. That’s what we are trying to achieve here with DataOps.
With new processes in place, becoming an expert on measuring your efforts will allow you and your team to fill gaps and find opportunities to drive innovation as you adopt these new methodologies.
Being able to measure the success you have in your data governance strategy is going to speed up your journey to becoming a data-driven organization.
If we are going to say that “data is the new oil”, we need to back that statement up by treating data as a product. Oil is constantly refined, but are you refining your data? Companies like Airbnb and Lyft are and that’s why they’re winning.
They did this by measuring their data efforts and finding a gap in the usage of their data stack, solidifying how important measurement is to data-driven organizations.
Enterprises waste millions of dollars on failed data initiatives because they apply outdated thinking to new data problems. This results in overly-complex, rigid processes that benefit the few and make the rest of us less productive.
Agile Data Governance is the fastest route to true, repeatable return on data investment. Use more of the knowledge you already have, make everyone smarter, and drive continuous improvement to keep your competitive edge. If you’re ready to take steps to get there, let’s get started!
Get a demo of our enterprise data catalog and see what it’s like to begin connecting your data sources and building your datasets for collaboration!
data.world makes it easy for everyone—not just the “data people”—to get clear, accurate, fast answers to any business question. Our cloud-native data catalog maps your siloed, distributed data to familiar and consistent business concepts, creating a unified body of knowledge anyone can find, understand, and use. data.world is an Austin-based Certified B Corporation and public benefit corporation and home to the world’s largest collaborative open data community.