PyTorch 2.0 Released: torch.compile and a Faster Backend

What happened

PyTorch 2.0 was released on March 15, 2023, according to the official release notes in the PyTorch GitHub repository, having been highlighted at the PyTorch Conference in December 2022. Composed of over 4,541 commits from 428 contributors since 1.13.1, it was described as a next-generation release that offers the same eager-mode development experience while fundamentally changing how PyTorch operates at the compiler level.

torch.compile

The main new API was torch.compile, which wraps a model and returns a compiled version. It is a fully additive and optional feature, so PyTorch 2.0 is 100% backward compatible by definition. Under the hood, torch.compile builds on new technologies including TorchDynamo, AOTAutograd, PrimTorch and TorchInductor, the last of which relies on OpenAI's Triton deep-learning compiler to generate performant GPU code on NVIDIA and AMD hardware.

Accelerated Transformers and MPS

PyTorch 2.0 shipped a stable version of Accelerated Transformers, offering high-performance training and inference through a custom kernel architecture for scaled dot product attention, exposed via a new scaled_dot_product_attention operator integrated with torch.compile. The Metal Performance Shaders (MPS) backend, providing GPU-accelerated training on Mac, expanded to cover over 300 operators.

Other changes

The release added functorch APIs in the torch.func module, optimized CPU inference on AWS Graviton3 instances, and shipped many prototype features across TensorParallel, DTensor and 2D parallelism. It also raised the minimum supported Python to 3.8 and dropped support for CUDA 10, and changed zero_grad to set gradients to None by default for better performance.

Why it mattered

PyTorch 2.0 was a landmark release that brought a compiler-based execution model to the world's most popular deep-learning framework without forcing developers to abandon eager mode. By making torch.compile a single, optional entry point to substantial speedups, it set the direction for the entire PyTorch 2 series.

Related on Skillo

See also: PyTorch 2.1's automatic dynamic shapes.

Sources

• PyTorch 2.0 release notes (GitHub)

Published date reflects the original event date (2023-03-15). This article is original Skillo editorial written from the sources above; facts were verified in September 2026.