Will Eaton

A high-throughput and memory-efficient inference and serving engine for LLMs

llm-d is a Kubernetes-native high-performance distributed LLM inference framework

Model Express is a Rust-based component meant to be placed next to existing model inference systems to speed up their startup times and improve overall performance.

Non-destructive GPU based idle-culler for RHOAI/Kubeflow workloads