Re-engineering the service that persists 1M+ sensor metrics a minute. Measure on production-shaped data first, then fix concurrency and write contention.
Tech lead · load testing, design and implementation
Stack
Java 21 · Spring Boot · PostgreSQL · Kafka · AWS
Ingestion pipelineValidated metrics from 1,000+ vessels1M+ / min
SinkTuned executors and ForkJoinPool; writes parallelised by metric category+85% throughput
PostgreSQLSeparate tables per metric categorysame cloud cost
Figure 1 — Simplified data flow
Situation
Sink is the service at the end of the ingestion pipeline: it takes metrics from 1,000+ vessels and persists them. As
volume climbed past 1M metrics a minute (~1.4B a day), Sink’s write throughput set the ceiling for the whole platform.
Every new vessel or sensor brought that ceiling closer.
Task
As tech lead, raise that ceiling with room for fleet growth, within two constraints:
No bigger bill. The answer had to come from the software, not from larger instances.
Live system. Client dashboards depend on fresh data, so changes had to roll out without disrupting ingestion.
Action
Reproduce before optimising. I built a load test that replays production-replicated data, so every change was
measured against the real traffic shape instead of synthetic payloads.
Right-size concurrency. I reworked executor and thread-pool configuration and ForkJoinPool usage to match the
actual mix of I/O and CPU work.
Remove write contention. I split writes by metric category into separate PostgreSQL tables and parallelised
them, so independent categories stopped competing for the same table.
Result
+85% pipeline throughput at the same cloud cost.
Headroom for fleet growth without scaling the infrastructure.
Hiring a Staff or Lead backend engineer?
I'm looking for a fully remote role on a B2B contract, with real ownership of architecture on data-heavy systems.