Data engineers are unhappy. Although that may seem like a broad, sweeping generalization, it’s true – and we have the stats to prove it. For our new white paper, Burned-out Data Engineers Are Calling for DataOps, we teamed up with DataKitchen to survey the job satisfaction of 600 data engineers and the results are striking: 97% are burnt out and 70% are planning to leave their current position within the next 12 months. Data engineers are so stressed out that 78% wish their job came with a therapist.
Yikes.
Solving this is no easy task and requires a detailed understanding of why data engineers are burnt out. Here are five factors contributing to their unhappiness.
1. Unreasonable requests
The role of data engineer is relatively new in the enterprise which means its nuances and complexities are not well understood. Data engineers can vouch for this given the number of unreasonable requests that cross their desk each week. Ninety-one percent of respondents reported receiving requests for analytics with unrealistic or unreasonable expectations – 61% said this happens “often” or “all the time.” These half-baked requests are a drain on resources and detract from the mission at hand.
91% of data engineers recieve requests for analytics with unrealistic or unreasonable expectations
2. Manual processes
A March 2020 survey by Gartner found that only 22% of data management teams’ time is spent on innovation. Instead, they’re focused on “operational execution,” i.e. implementing and maintaining production initiatives. In other words, we’re asking highly skilled data team members to manually execute procedures that ingest, clean, transform, and disseminate data. Our survey confirms the magnitude of this problem, with 50% citing manual processes as a top issue impacting data engineers.
50% of data engineers cite focusing too much time on manual work as a primary driver of burnout
3. Finding and fixing errors
Maintaining data quality is an important part of the job for data engineers, but when the flow of errors is relentless, finding and fixing those errors can take on a life of its own. Fifty percent of data engineers surveyed said they focus too much time on this, which limits their ability to work on more impactful projects. A typical enterprise experiences multiple data, pipeline, or analytics errors per week. Managing a continuing succession of outages while trying to keep development projects on schedule and under budget is exceedingly difficult if not impossible.
50% of data engineers say they spend too much time finding and fixing errors
4. Shouldering blame
When something goes wrong with a company’s data and analytics, no one person should shoulder the blame, but according to survey results, that happens 87% of the time, and data engineers are tired of it. These negative feelings can manifest as anxiety and a reluctance to take technical risks – a significant obstacle to productivity and innovation.
87% of data engineers say they are blamed when something goes wrong with a company's data and analytics
5. Overly restrictive governance
Data engineers entered the field because they want to work with data, but 69% say restrictive governance processes make that difficult to do. The “lock-it-down” approach employed by many organizations lacks transparency, often resulting in more work for data engineers who are responsible for following complicated processes to access essential data sources.
69% of data engineers say their company's data governance processes make their job harder
Make no mistake, the situation is dire, but there are steps you can take to improve the employee experience for data engineers. In our webinar, The struggle is real: 10 tips to overcome data engineer burnout, data.world CTO Bryon Jacob and DataKitchen CEO Chris Bergh dive deeper into the factors that contribute to data engineer burnout and offer actions you can take today to make life better for these valuable employees.
