Distributed LLM Inference Platform

Multi-GPU LLM serving: one model worker per GPU, with a load balancer that routes each request to the least-busy worker.

Work in progress. Throughput and latency benchmarks will be published here.

Code: server/worker.py · server/load_balancer.py