We've pulled a lot of SD cards out of a lot of loggers over the years. Usually it's because the cell connection dropped and somebody wants the missing week back. Every so often, though, you load the card next to the data that did make it through the API, and the two don't quite match.
I don't mean missing rows. I mean different numbers for the same minute.
Sometimes the gateway is rounding. Sometimes the timestamps are an hour off because something in the chain handled daylight saving time differently than the logger did. Sometimes a cloud platform somewhere along the way is "cleaning" the data with an algorithm nobody told you about. On the plant side it's the same story with different names: the SCADA screen says one thing, the historian archived something slightly different, and the historian's compression settings turned a slow drift into a staircase made of rounding errors.
Which one is right? It depends on what you need the data for. You can't even ask the question, though, unless you can put both copies side by side and know exactly which one came from where. Who has time for that?
That's a big part of why Datasparks has its own target schema. When you bring data in, we ask you to tell us what's what. One of the things we ask is where it came from.
What years of sensor data taught me
I've spent most of my career working with data, and the hard part has almost always been the variety. Every instrument vendor has their own idea of what to call a site, a sensor, or a measurement. So does every firmware version, and every integrator who touched the system after that. I've lost count of the projects where the same station showed up as "Station 1", "STN_01", and "station-one" in three different files.
Cleaning, normalizing, harmonizing, comparing. Every one of those starts with the same question: what exactly is this number? You can't apply a sensor-specific quality rule if you don't know which sensor it came from. You can't average a 1-second reading together with a 15-minute average (well, you can, people do it all the time, but the result doesn't mean anything). When the identifiers are clean, most of this work is just arithmetic. When they're buried in free text, most jobs start with writing a parser, and the next file usually needs a slightly different one.
A few years ago I wrote a set of data standards recommendations for a state air regulator. One of them was to record the technologies that collected, aggregated, and managed each piece of data, so the data could be flagged if a particular platform was known to adjust it with an algorithm. That idea stuck with me. In Datasparks that idea lives on the source.
In the same report I recommended packing the aggregation chain into the metric name, something like 1h avg(1m avg(1s relative humidity percent)), so an analyst could read the whole history of the number from its label. I'd do that one differently now. It made sense when the analyst was a person reading a spreadsheet. A computer can filter and validate a field directly. A name it has to parse, and every vendor's names parse a little differently.
Tags are great, until they're not
If you come from the plant floor, you're probably thinking we already solved this. It's called a tag.
A lot of plants organize their equipment around ISA-95, the standard that describes how plant-floor systems connect to the business side. It lays out a hierarchy that goes enterprise, site, area, and then down into the work centers and units inside each area. Tag naming conventions often follow that hierarchy, so you end up with something like PLANT1.AREA2.PUMP3.DISCHARGE_PRESS.PV. Plant, area, equipment, measurement, and which value of it, all in one string.
For what tags are built to do, they're excellent. An operator and a control system need to agree on exactly one point, and the tag does that. The trouble starts when you want to ask questions across the hierarchy. Discharge pressure on every pump in every plant? Everything in Area 2? Everything that the historian recorded versus everything that SCADA recorded? Now you're writing string-splitting code. It works great until you hit the tag somebody named by hand years ago that doesn't follow the convention.
It's a bit like writing someone's entire mailing address in the "name" box on a form. The mail still gets there. Good luck sorting it by zip code.
So we split the tag up
Datasparks keeps each piece in its own field. The source is the system or feed that the reading came through. The sampling node is where it was measured. The metric is what was measured. If you harmonize the data later, say to 15-minute averages, that version gets its own variant so it never overwrites what was actually observed. A dataset in Datasparks is one specific combination of those four.
To be clear, Datasparks doesn't parse your tags for you. You do that once, when you map a file. This column is the location, this one is the measurement, this one is the value. Or you set a default source and sampling node for everything a connector brings in. Anything new that Datasparks creates from your data sits as provisional until someone accepts it or merges it into something you already have. It's a little work up front (and we're working on giving you AI help for that). It's a lot less work than trying to remember what "STN_01" meant two years from now.
Why the source field matters so much
Most of the tools I've used don't have a source field at all. They treat "temperature at Station A" as one series, and the SD card and the API are just two ways of filling it in. So when the second copy shows up, it usually overwrites the first or gets thrown out as a duplicate, and the difference you actually needed to see is gone.
In Datasparks the source is part of what makes a reading unique. Map the SD card and the API as two different sources and you get two separate records for the same minute, and nothing collides. The SCADA export and the historian archive work the same way: same sampling node, same metric, different source.
Want to compare them? Build a data product spread by source so each one gets its own column, then harmonize it to a common interval so the timestamps line up. Then just look, we already ran the comparison stats for you during the data product build. If the deadband turned a slow drift into a staircase, you'll see it. If something botched daylight saving time, you'll see a whole stretch of readings shifted by an hour.
Now you get to decide what to do about it, with both numbers in front of you. Maybe the API is fine for the live dashboard and the SD card is what goes in the quarterly report. Maybe the historian's compression settings need to change. That's your call. The point is that you get to make it before a regulator asks why your numbers don't match the logger.
Yes, it costs something
Somebody has to do the mapping. A weird file takes longer to bring into Datasparks than it would to dump into a data swamp that'll take anything. Some data doesn't fit neatly on the first try. A sensor bolted to a truck, for example, needs to be set up as a mobile sampling node instead of a fixed site, and people don't always know that going in. (Again, we're working on AI help for that!)
I'm biased, since I run the company that built this. I've also done the cleanup the other way more times than I'd like. The mapping has to happen at some point. Up front, it's just mapping. Later, it's mapping plus figuring out everything that went sideways in the meantime.
Whatever tools you end up using, make sure you can answer four things about every reading before you start analyzing it: where it was measured, what was measured, how it got to you, and what's been done to it since. "How it got to you" is the one I see skipped most often. It's usually the one that would have told you something changed along the way. And unknown changes are usually the cause for lost trust with your stakeholders.
