Nov 2025 to now · Serving under load · Federal ID-verification platform (ICP)
From about one request at a time to about a hundred.
A government ID-check service built from 15+ AI models could serve one request at a time on each CPU-only node. I traced the cause to a MediaPipe/TFLite step and moved that path to ONNX Runtime. A single node now handles about a hundred requests in flight.
- My part
- Found the bottleneck and migrated that path to ONNX Runtime.
- Team
- Technical lead on a team of 5 or more engineers.
- Result
- ~1 → ~100 concurrent requests per CPU-only node.
- Not stated
- Latency, request payload and load-test method.