The bucket was never the problem
Every team we talk to has already tried the bigger bucket. The new data engineer, more storage, more compute, new libraries, a time-series database, then a lakehouse. And they still can't adapt quickly to test a new hypothesis, add quality checks, identify issues in the data, or try a new data exploration tactic. That should tell you something. The whole stack was sold on one premise: that the hard part is volume, that if your data hurts it must be because there's so much of it.
In our experience, that premise is quietly wrong for most of the people who feel the pain. The biggest challenge we've run into, over and over, was never the volume of the data. It was the variety of it. And variety doesn't get better when you buy a new expert for the team or a bigger bucket. It gets worse, because now you have more disagreements sitting in one place, and you thought the hard part was already solved.
I want to make the case that most measurement projects don't have a big-data problem. They have a harmonization problem wearing a big-data costume.
A quick financial analogy, because I have one for everything
Imagine you run a company where every department keeps its own books. North America records revenue in dollars, your European team in euros, the field crew logs things in "a truck full" and "about half a day." Every department uses a different chart of accounts and a different fiscal calendar — one closes on the 30th, one on the last Friday, one whenever the manager gets around to it.
Now your board asks a simple question: what did we actually earn last quarter?
You do not have a volume problem. You could have three transactions or three billion; the wall is the same. You cannot add anything up because nothing agrees. There's no common currency, no common ledger, no common calendar. The work isn't storing the numbers. The work is getting them to agree well enough that a sum means something.
That's what harmonization is. And measurement data (sensors, loggers, field instruments, lab exports) is a business where every department kept its own books and nobody was ever asked to reconcile.
Three things that don't agree
When we look at why two datasets that "measure the same thing" won't combine, it almost always comes down to three disagreements. None of them is about size.
They disagree on names. One logger exports a column called Temperature. The next vendor's export calls it T_degC. A third writes Temp and puts the units in a second header row. To a human these are obviously the same measurement (at least until you question whether it's F or C). To a machine they're three unrelated columns, and any tool that joins on column name will happily give you three half-empty datasets instead of one full one.
They disagree on clocks. Timestamps are the single hardest column in almost every dataset I've seen. One file is in UTC, one is in local time with no offset written down (and "local" only resolves against a time-zone database), one splits the date and the time into separate columns, one stores an epoch integer, and one uses 03/05/2021 — which is March 5th or May 3rd depending on which side of an ocean the instrument was configured on. One provides an hourly average timestamp at the start of the hour, another at the end of the hour. Every one of those is "the time." None of them lines up with the others until someone decides what they really mean.
They disagree on cadence. One sensor reports every second, one every fifteen minutes, one only when a value crosses a threshold, and one whenever the field tech remembered to walk out and download it. You cannot (should not... we all know 'that guy') put a one-second stream and a fifteen-minute stream on the same chart, or in the same average, until they share a grid. And the moment you make them share a grid, you've made a decision — which we'll come back to, because it's the honest part.
Names, clocks, cadence. That's the whole fight. Volume is not on the list.
Getting the names to agree
The fix for the names problem is boring and that's exactly why it works: you say, once, that Temperature and Temp and T_degC all mean the same measurement. Each vendor's column is aliased to the same metric, so you declare the equivalence a single time and the instrument's naming quirks stop leaking into your analysis from then on. It's the chart-of-accounts move: agree once on what to call a thing, and every department's books suddenly add up.
The part I'd push you on, whether or not you use Datasparks.io, is that this agreement should be recorded, not performed by hand in a spreadsheet every quarter. A rename you do in Excel or hardcode into a messy python notebook is invisible the next time a file lands. A mapping you declare once is a durable statement about what your data means. One is a chore you repeat forever; the other is an asset that compounds.
Fallbacks and overrides: a vocabulary for messy sources
Here's a small distinction that turned out to matter more than we expected. When a source column is missing or ambiguous, there are two very different things you might want to say, and most tools blur them into one.
A fallback is a hypothesis: "if this field is blank, assume the interval is 15 minutes, but check me against the data, and if the evidence disagrees, believe the evidence." An override is a decision: "this value is 15 minutes, full stop, don't reconcile, I know something the file doesn't."
Back to the books: a fallback is the accountant's pencilled-in estimate before the invoice lands; an override is the figure on the signed, closed ledger. Same number, completely different level of commitment. You want your system to know which is which.
A fallback and an override are not the same claim, and pretending they are is how bad numbers sneak in wearing a confident face. A fallback is you being honest that you're guessing. An override is you taking responsibility. Giving each its own word, and letting the pipeline treat a guess as a guess and a decision as a decision, is a small piece of vocabulary that keeps assumption from quietly masquerading as measurement.
Getting the cadence to agree is a modeling decision, and you should say so
This is the part most "just clean it up" workflows skip, so I'll be blunt about it.
The moment you resample a one-second stream down to one-minute values, or fill a gap in a fifteen-minute stream, you are no longer holding raw truth. You're holding a model. You chose how to average (a time-weighted mean is not the same as a last-value-wins), you chose how far to interpolate across a gap before you give up and leave it empty, you chose what "one minute" even means. It's the fiscal-calendar problem one more time... deciding when a minute closes is as much a choice as deciding when a quarter closes, and it moves the number. Those are real decisions with real consequences for the answer.
The wrong response is to pretend the choice didn't happen. The right response, the one I'll defend, is to make the choice explicit and write it down next to the data, so anyone can see the stance you took.
Concretely, on our platform a harmonized dataset carries a code that records exactly that stance, something like 1m-pchip-twm-tg30-h15-nmc-pg60. You don't have to memorize it; the point is that it's legible. It says: one-minute grid, PCHIP interpolation going up, time-weighted mean going down, gaps up to thirty seconds get filled and larger ones are left honestly empty. Change any of those and it's a different code, because it's a different model. Nothing is "just cleaned." Every transformed number can point back at the decision that produced it.
I care about this because it's the same principle as the fallback-versus-override distinction, but one level up: the goal is never to hide the assumptions, it's to make them measured, named, and defensible. When a regulator, a client, or your own future self asks "how was this number derived?" you should have an answer that isn't a shrug.
Even location has to agree now
The newest version of this problem is spatial. Put sensors on trucks, drones, or people and every measurement now happens somewhere, and the somewhere disagrees just like everything else — different GPS sources, different position measurement rates than the sensors they ride with, chipset or processing latency that reports a position a couple of seconds after the moment it describes. It's one more set of books nobody reconciled.
So we harmonize position the same way we harmonize everything else: align each reading to where the platform actually was at that instant, on a common spatial grid, and, this is the honest part again, mark the rows where there was no trustworthy position as an honest gap rather than inventing one. A modeled path is a stance you chose, recorded like any other. It is never a "trust me, it was probably here."
The world outran the premise
None of this is a knock on big-data tooling. Big data management tools are genuinely good at what they were built for. The problem is that "big data" became the default diagnosis for every data headache, and for the people I talk to (field teams, consultants, monitoring program managers, researchers) it's usually the wrong one. They were sold a bigger bucket to solve a translation problem. The bucket arrived. The translation problem is still sitting there.
My honest take, and you're welcome to disagree (I think that's great, actually) is that the useful question in front of most teams isn't "how do we store more?" It's "how do we get our data to agree?" Agree on names, agree on clocks, agree on cadence, and record every modeling decision you make along the way so it can be defended instead of just trusted. Do that and a surprising amount of the "we need a data platform" pressure turns out to have been a harmonization problem the whole time.
Your data probably isn't too big. It's just never been introduced to itself.
If you want to see what "recorded, defensible harmonization" looks like on a real dataset, the fastest way is to put a couple of your own mismatched exports through a trial project and watch the names, clocks, and cadence come into agreement — with the decisions written down, not hidden.
