Serving systems
8 papers in this thread, across 3 domains.
Progress0 of 8
- 2024NanoFlow: Towards Optimal Large Language Model Serving ThroughputServing2408.12757Aug 22, 2024~108 minServing
- 2024Llumnix: Dynamic Scheduling for Large Language Model ServingServing2406.03243Jun 5, 2024~113 minServing
- 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM ServingServing2407.00079Jun 24, 2024~127 minServing
- 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-ServeServing2403.02310Mar 4, 2024~86 minServing
- 2023SGLang: Efficient Execution of Structured Language Model ProgramsAgents2312.07104Dec 12, 2023~101 minAgents
- 2023S-LORA: SERVING THOUSANDS OF CONCURRENT LORA ADAPTERSServing2311.03285Nov 6, 2023~106 minServing
- 2023PUNICA: MULTI-TENANT LORA SERVINGServing2310.18547Oct 28, 2023~100 minServing
- 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked PrefillsLLM Systems2308.16369Aug 31, 2023~105 minLLM Systems