Data engineers — The reluctant heroes of your data team
Does Superman ever feel burned out? Does he feel he has to constantly meet unrealistic expectations? Does he resent the lack of work-life balance that comes with the title “hero”?
And if so, is he likely to hang up his cape and “join the ranks of the Great Resignation”?
Becoming a workplace “hero” often means trading too-much personal time for professional success. And while colleagues who go above and beyond to meet deadlines or solve business problems are celebrated in the moment, the amount of overtime required to meet unrealistic expectations frequently leads to burnout, anxiety, and even depression.
As we’ve seen over the past two years, many workers have had enough, with record numbers of Americans quitting their jobs. And there’s no sign of the trend slowing down.
Some jobs are more susceptible to burnout than others. And unfortunately for data-driven organizations, data engineers are among the professionals suffering most.
Data engineer kryptonite — Unreasonable expectations
Data engineers are the backbone of the modern data-driven enterprise, and their work ensuring that data is available, secure, correct, and fit for purpose is absolutely critical to a data team’s success
But according to a recent survey jointly performed by data.world and DataKitchen, 97% of data engineers report burnout in their day-to-day jobs. Because of that, 70% of data engineers say they are likely to leave their current company for another data engineering job in the next 12 months. And if that’s not enough to illustrate their professional misery, 79% have considered leaving the industry entirely. Given today's increasingly competitive talent market, losing a high-performing data engineer could be crippling for any data team.
According to our survey, data engineers often find themselves spending too much time finding and fixing errors, performing manual and repetitive processes, and dealing with an endless barrage of unreasonable requests from colleagues.
In the face of these data governance challenges, data engineers are considered “heroes” when they meet their deliverables. But these heroes give up work-life balance, and the sacrifices of a company’s heroes are quickly forgotten when there is a new deliverable to meet. The type of heroism that requires “going above and beyond” is virtually impossible to sustain over a long period; just ask Clark Kent. And it ultimately just resets expectations at a higher level without addressing the root cause of burnout-inducing organizational failures.
Three tips for helping your data engineers reclaim their work-life balance
Given the alarming statistics above, it’s clear that organizations and their data engineers need to work together to foster pragmatic and successful working environments that support big-data environments.
That’s why I’m suggesting a few simple tips to help organizations establish achievable data project standards and goals, and at the same time, keep their data engineers productive, engaged, and happy.
1. Set your data engineers up to succeed
Any organization with a data engineer - let alone a data engineering team - should have a complete understanding of that team’s capabilities. This will help ensure that any requests made of that team are both realistic and fair.
Another way to make your data engineers’ lives easier is to make sure that they have clean, accurate data at their fingertips, and an effective way to ensure that’s what your engineers are working with is to catalog enterprise data. By implementing a data catalog platform, data engineering teams can better understand and connect all data sources, simplifying the management and monitoring of your data pipelines.
2. Invest in automation (and cut down on bureaucracy)
One of the most common sources of burnout is a mountain of menial tasks. Crucially important processes like source control and software versioning can be repetitive and dull. And continually running tests to ensure data quality can eventually bore even the most dedicated engineer. By adopting software automation, you not only save your engineers from these mind-numbing practices; you free them up to work on what they signed up for in the first place — uncovering valuable data, creating innovative analytics, and helping your business thrive.
3. Practice Agile Data Governance
On the topic of working with data in our study, 69% of those surveyed said their company’s data governance policies make their day-to-day job more difficult. The “lock-it-down” approach employed by many organizations lacks transparency, often resulting in more work for data engineers who are beholden to complicated processes for managing access to data sources.
Enterprises can alleviate this burden by practicing Agile Data Governance. Unlike traditional top-down data governance, Agile Data Governance opens up some historically restricted governance functions to a broader audience to iteratively capture the knowledge of data producers and consumers so everyone can benefit. Think of it like putting access on rails: making data fully auditable and predictable simplifies its management, so data engineers are free to work on more impactful projects.
Constantly having to exert an heroic effort just to get through your work day can be exhausting and overwhelming. By adopting the three above tips, you'll empower your data engineers to get back to what they want to do and let leave leave their cape at home.
Learn more about Agile Data Governance by reading our Agile Data Governance Playbook.
