📚 Resources

• vLLM Paged Attention kernel-level design: https://docs.vllm.ai/en/latest/design/paged_attention/
• vLLM distributed/parallelism docs (batching-relevant scheduling): https://docs.vllm.ai/en/stable/serving/parallelism_scaling/
• PagedAttention: OS-style paging for the KV cache: https://medium.com/@abonia/vllm-and-pagedattention-a-comprehensive-overview-20046d8d0c61
• Continuous (iteration-level) batching vs static batching: https://insujang.github.io/2024-01-07/llm-inference-continuous-batching-and-pagedattention/

Week 05 · Hugging Face TGI and SGLang Serving Engines

Notion ID: 392c08fd-0278-81cf-be78-e0126954ee66 Notion URL: https://app.notion.com/p/392c08fd027881cfbe78e0126954ee66