# Real-time ML Serving
DATE: Unknown Date
Back to Blog
Engineering • Jul 05, 2025
# Real-time ML Serving: The 50ms Barrier
In 2018, before "LLM" was a household acronym, serving machine learning models in real-time (< 50ms
latency) was considered a dark art. Most companies ran batch jobs overnight. But fraud happens in
milliseconds, and recommendations need to happen before the user scrolls past. We had to build a
system that could serve inference at the speed of a database query.
## Optimizing the Runtime
Python is great for training, but it's terrible for serving. The Global Interpreter Lock (GIL) is
a bottleneck for high-concurrency workloads. We moved our heavy lifting to C++ using ONNX
Runtime. This allowed us to export PyTorch models into a hardware-agnostic intermediate
representation that could be executed incredibly fast.
## The Feature Store Problem
The hardest part of real-time ML isn't the model; it's the data. You need to calculate "features"
(e.g., "number of clicks in the last 5 minutes") instantly. If your training data comes from a
data warehouse (Snowflake) but your inference data comes from a cache (Redis), you have
Training-Serving Skew. Your model will fail in production because the data
looks "different."
We solved this by building a unified Feature Store. Definitions were written once in code, and
the system automatically populated both the offline warehouse (for training) and the online
low-latency store (for inference). This ensured mathematical consistency between the lab and the
real world.
## Conclusion
Performance is a feature. If your AI feature takes 2 seconds to load, users won't use it. You
have to be faster than thought.