TL;DR – Imagine this future – Data Engineering aims to help businesses to deal with data. Data Engineers are not doers anymore, they are enablers.
Love it? Then keep reading.
———
It has been said so many times about Data Engineering, what this is all about, and how it develops… Google it, and you will get something around 1,750,000,000 results (at least, I got that many results at the time of this article).
No doubts, there is little to nothing uncovered or unclear about the topic.
However, some aspects of Data Engineering still might be arguable or too-broad-to-accept. Anecdotally, the meaning or the purpose of Data Engineers is one of these things.
I met many definitions of the purpose of Data Engineering, and every time these were about certain technical tasks Data Engineers should perform. The range of these tasks is wide: from more generic “… they handle everything related to data” to more specific like “… bring data from point A to point B…”. This all is about Data Engineers being “doers”.
To be honest, I followed this approach for a while until I met a great article from Chris Riccomini “The Future of Data Engineering”. I realized the real purpose of Data Engineers nowadays Enterprise is “to help an organization move and process data”.
Do you recognize the difference? It is not about “to provide” or “to move/process” but “to help”. This way, Data Engineers become “enablers” rather than just “doers”.
Some time ago, everything IT-related was sacred knowledge, and a comparable small group of IT wizards could work with data adding value to the company.
IT departments played a mystic “black box” role where businesses had to put their data and blindly trust IT folks who performed their magic. Restricted access to the specific knowledge and immaturity of tools caged data-related activity within the IT department.
Everything has changed since that time.
Nowadays, the line between Business and IT is blurred if not completely vanished, and we may discover how these domains are tightly integrated.
Market demands more and more products, here and now, and having knowledge caged within one team becomes very unproductive and expensive.
Complex and comprehensive tools become available to non-technical people. Look around, and you will see tons of the products that do out-of-the-box data integration, data visualization, machine learning.
These tools plus tech-savvy business folks empowered Business with the ability to do data processing within the business unit. “Shadow IT” is not an underground anymore, it claims the right to deal with own data.
Of course, transformation requires time and effort, and Data Engineers may need to keep doing things in the traditional way, but with the new paradigm in mind.
So… What is next?
Data Engineers should start building frameworks and automation, and help Business to use these tools.
Data Engineers should change the way they help Business. #Knowledge sharing, #building frameworks, #enabling capabilities, #scaling — these are new buzzwords, and this is a fresh course to follow.
Everything has changed. So does Data Engineering.
Take the next step.
Recent Comments