Developers · Updated 20 Oct, 10:19 pm IST
Google launches native vLLM and SGLang PyTorch engines on TPU
Why it matters for readers: It’s a development that makes it easier to run advanced model servers on Google’s TPUs with familiar PyTorch tools.
- Google introduced TorchTPU, which makes TPU a native PyTorch device so tensors are standard PyTorch tensors on a tpu device and code can run eagerly or via torch.compile using XLA.1
- New vLLM and SGLang inference engines were built on top of TorchTPU to provide native PyTorch backends for those serving stacks.1
- vLLM was developed in partnership with Inferact and SGLang was developed in partnership with RadixArk, according to Google.1
- The engines ship with optimized, production-ready serving features and custom Pallas kernels as part of an integrated TPU inference stack.1
Get a brief like this every morning
Uzha reads hundreds of sources and gives you the stories that matter for your work, with every source linked. Free.
Get started