Serving and Inference (Rewrite with Code)
摘要
Production inference is where throughput, latency, and correctness meet real users. Keep the stack small and deliberate: a serving runner (TorchServe, BentoML, or Ray Serve), a streaming edge (FastAPI), an LLM back end (vLLM or TGI), and – when it truly fits – serverless.