For a decade, the reflexive answer to "we have a lot of sensor data" has been either: ad-hoc partial analysis in Excel or a python notebook, or a distributed cluster in Spark or one of the platforms wrapping it. That reflex made sense when it formed. I want to walk through why, for the data sizes a sensor network actually produces, the reflex is now costing teams five to fifty times more than the job requires and exactly where the reflex is still right.
A note on method before the numbers, because the title says "we ran the numbers" and I want to be precise about which numbers are whose. The cost model and the survey are ours; the benchmarks are deliberately not. Everything load-bearing below is published, third-party, and linked, because "vendor benchmarks own product" is a genre nobody should trust.
How big is sensor data, really?
Start with the premise, because everything follows from it. A dozen loggers reporting every minute produce several million rows a year. That translates to single-digit gigabytes. A serious regional network with high-frequency instruments might process tens of gigabytes in its largest jobs. The modal ingestion job in our world is small enough to fit in one machine's memory with room to spare, and it has been for years, quietly, while "big data" kept its grip on the architecture conversation. The hard part of sensor data was never its volume; it's the variety. The mismatched vendors, clocks, and vocabularies. Volume is the one problem we mostly don't have.
Meanwhile, single machines got enormous, and a class of engines emerged to exploit them, DuckDB chief among them, that process data in columns, in bulk, in memory, with none of the machinery that distributed compute requires. The question "one strong machine or a cluster of them?" now has a measurable answer at every job size.
The mobilization fee
Here's the analogy that makes the economics intuitive. Hiring a distributed cluster is like hiring a road crew: for a highway, magnificent; the mobilization is the price of capability. But call the same crew for a pothole and you'll pay the mobilization fee for the trucks, the cones, the crew assembling anyway for a job one person with a bag of asphalt could finish before the trucks arrive.
Spark's mobilization fee breaks into three tolls, and none of them is a bug. Each is the honest cost of hiring the crew:
Toll one: spin-up. A classic cluster takes five to twelve minutes to provision and start. This is the cost of the trucks and cones arriving before anyone touches asphalt. A five-minute job pays a hundred percent overhead before its first row; the toll only amortizes on jobs long enough to dwarf it.
Toll two: double billing. On the managed platforms the crew's day rate gets charged twice: once for the underlying cloud machines, and again in the platform's own compute units on top. The meter multiplies the trucks instead of replacing them.
Toll three: coordination overhead. A crew spends part of every day coordinating itself, and a cluster is no different: serialization, shuffling data between nodes, and the JVM machinery (the Java runtime overhead Spark carries by default). This is work that exists only because the job was split up. When the data could fit into one machine's memory, every one of those cycles is spent solving a problem you didn't have.
What the published measurements say
The cleanest apples-to-apples numbers I've found come from Walmart Global Tech's engineering benchmark, on their own 8-30 GB workloads that fit squarely into the sensor-network size class. Aggregations: 41× faster on DuckDB. Joins: 13–16× faster. And the infrastructure bill underneath: $1.02 an hour for the single machine against $55.64 an hour for an 11-node cluster, roughly 54× more cost to lose those races. Speed and cost compound: the cheaper machine also finishes first. Coiled's TPC-H benchmarking, a standard analytics benchmark suite, tells the same story across the sub-100 GB range.
Where the reflex is still right
Now the other half, with the same confidence, because this is where a fair post earns the right to its first half.
The crossover is real and starts around 100 GB. Miles Cole's benchmarking found Spark winning on cost-efficiency at 100 GB per job (the largest scale he tested) once the machines reach eight-plus vCores (roughly an eight-core cloud box). Our reading of the broader evidence: from there up to a terabyte the two approaches converge toward parity, and above a terabyte, distribute. That band is a synthesis, not a single benchmark, and I'd take a loose read on it.
Cardinality can beat you long before volume does. In a documented record-linkage thread, a join generating 3.3 billion candidate pairs ran a 500,000-record deduplication out of memory on a machine with two terabytes of RAM, while the same practitioner's 9-node Spark cluster handled a larger 11-million-record run in three hours. A join's intermediate result, not its input, is what has to fit; when comparisons explode combinatorially, no single box saves you.
Single nodes concede resilience and concurrency. A twelve-hour job on one machine restarts from zero when the machine dies; Spark's fault tolerance is precisely the machinery for surviving that. And one machine serves one working set well. A large team hammering concurrent heavy queries eventually wants distribution for reasons that have nothing to do with any single job's size.
If your workloads live in those paragraphs, use Spark, sincerely. The argument here was never "single node always wins." It's that the crossover sits around two orders of magnitude above the modal sensor-network job, and defaults should serve the modal case.
What we actually run
We put this conclusion into production rather than just a blog post. DuckDB does the heavy lifting in the Datasparks ingestion pipeline; parsing, harmonizing, and writing sensor data into an Iceberg-compatible lake on hardware we own rather than rented cluster-hours, which is why cluster spin-up appears nowhere on our public meter. And a second DuckDB, compiled to WebAssembly, runs inside your browser. After the first sync, exploratory queries execute on your own machine without a network round-trip, which is what makes exploration feel like exploration. One engine philosophy, both ends of the pipe. One should match the machinery to the actual size of the job. When you're ready to see how this engine choice plays out across whole platforms rather than benchmarks, the lakehouse comparison picks the story up from here.
And here's the satisfying part; you can check the premise yourself in an afternoon, no procurement required. DuckDB is free and installs in a minute. Take your largest real month of data (the actual files, not a synthetic sample) and run your gnarliest aggregation on a laptop. Time it. Then look up what the same workload costs on your current metered platform, or on the quote you're evaluating. That single experiment settles, for your data rather than Walmart's, which side of the crossover you live on. Given that cluster spin-up alone runs five-plus minutes, don't be surprised if the laptop finishes before a cluster would have booted.
The generalized principle, since that's how I'd want any vendor to leave it: before you architect for the data you're told to expect, measure the jobs you actually run. If your biggest month fits in one machine's memory, and for most environmental or industrial programs it does, several times over, then the cluster isn't insurance, it's a mobilization fee on every pothole. The savings aren't in a smarter contract with the road crew. They're in noticing you never needed one.
