In 2018, before "LLM" was a household acronym, real-time inference (< 50ms latency) was still unusual outside card fraud and ad tech. But fraud happens in milliseconds, and recommendations need to happen before the user scrolls past. We had to build a system that could serve inference at the speed of a database query.
Optimizing the Runtime
Python is great for training, but it's terrible for serving. The Global Interpreter Lock (GIL) is a bottleneck for high-concurrency workloads. We moved the heavy lifting into a custom serving layer, using C++ bindings for our Python models, built to execute incredibly fast.
The Feature Store Problem
The hardest part of real-time ML isn't the model. It's the data. You need to calculate "features" (e.g., "number of clicks in the last 5 minutes") instantly. If your training data comes from a data warehouse (Snowflake) but your inference data comes from a cache (Redis), you have Training-Serving Skew. Your model will fail in production because the data looks "different."
We solved this by building a unified Feature Store. Definitions were written once in code, and the system automatically populated both the offline warehouse (for training) and the online low-latency store (for inference). This ensured mathematical consistency between the lab and the real world.
Conclusion
Performance is a feature. If your AI feature takes 2 seconds to load, users won't use it. You have to be faster than thought.