Do you need data engineering, or do you need one spreadsheet to stop being retyped?

Data engineering is the work of moving data from where it is created to where it is used, reliably and without a person in the middle. Pipelines that extract and load, somewhere to put the result, and the definitions that decide what a phrase like active customer means. That is the discipline in a sentence.

What it does not solve is disagreement about what the numbers mean. If sales and finance produce different revenue figures because they count differently, a pipeline will deliver both numbers faster and the argument will continue unchanged. The definition is the problem, and no amount of infrastructure fixes a definition.

This matters because the two get confused constantly. A team buys a platform to settle a disagreement that was never technical, then concludes the platform failed.

Count the hands. Take one number that matters, the one quoted in a board pack or a weekly review, and trace it backwards. How many systems did it come from, how many times was it copied or retyped, and how many people touched it before anyone was willing to say it out loud.

One system and one export is a reporting problem. Two systems and a person reconciling them is an integration problem. Four or more sources, a recurring manual merge, and definitions that live in one person’s head is where data engineering starts to earn its cost.

The volume of data is close to irrelevant at this scale. Teams assume they need a warehouse because they have a lot of rows. What justifies infrastructure is the number of joins a human performs by hand every month, not the row count.

Published figures put ongoing maintenance of custom software at roughly 15 to 25 percent of the original build cost per year, and data infrastructure sits at the harder end of that range because it has a permanent relationship with every system feeding it. Source systems change their fields without telling you. APIs are versioned. A vendor is replaced and a pipeline stops quietly.

The cost that surprises people is not the build. It is the second year, once the people who built it have moved on and nobody remaining can say why a particular transformation exists.

So the honest question before commissioning any of this is not what it costs to build. It is who maintains it in year two, and whether that person knows yet.

When your sources are common ones and your questions are common ones. If the data sits in a handful of mainstream systems and you want standard reporting on top of it, products exist that do this and they will be cheaper and faster than anything built for you. Paying to own a problem somebody else has already solved is the most common error in this category.

Building becomes defensible when the shape of your data is genuinely unusual, when the logic that turns raw records into a meaningful number is specific to how you operate, or when the thing being measured is itself part of what makes the business work.

The third option gets left out of most comparisons and is often the right one: buy the platform, and build only the transformation layer that is actually yours.

Delete a report. Take the recurring reports your team produces and ask who read the last one, and what they did differently because of it. A meaningful share of reporting load exists through inertia, and removing it costs nothing.

Then write down the definition of the three numbers that matter most and ask two people in different functions to check them. If they disagree, you have found work that no infrastructure will do for you.

Only then is it worth asking anyone for a number, because a decision made at that point is about a real gap rather than a general sense that the data situation is bad.

Teams under about ten people with one source system. The honest answer there is usually a better spreadsheet and a convention about who owns it.

Teams whose real problem is that one person leaves and the knowledge goes with them. That is a documentation and ownership problem wearing a technical costume.

And anyone whose motivation is that building it sounds more satisfying than buying it. That is a real force in these decisions, and worth naming, because it is often the one doing the deciding.