PyTorch 2.0 Release

PyTorch 2.0 release accelerates open-source machine learning

Hey Guys,

Just a quick note, Meta is releasing PyTorch 2.0.

Yann LeCun @ylecun

PyTorch 2.0 is announced! Main new feature: cmodel = torch.compile(model) Faster training with no code modification. Available in nightly build. Stable release scheduled for early March

PyTorch @PyTorch

We just introduced PyTorch 2.0 at the #PyTorchConference, introducing torch.compile! Available in the nightlies today, stable release Early March 2023. Read the full post: https://t.co/DFO0HojkMt 🧵below! 1/5 https://t.co/ejEQsJHuB2

PyTorch Important for Machine Learning

As you know, among the most widely used machine learning (ML) technologies today is the open-source PyTorch framework.

It’s becoming pretty dominant. PyTorch got its start at Facebook in 2016 with the 1.0 release debuting in 2018. It’s come pretty far in just five years.

In September 2022, Meta moved the PyTorch project to the new PyTorch Foundation, which is operated by the Linux Foundation. Today, December 2nd, 2022, PyTorch developers took the next major step forward for PyTorch, announcing the first experimental release of PyTorch 2.0.

So it looks like the new release promises to help accelerate ML training and development, while still maintaining backward-compatibility with existing PyTorch application code.

To complement the PyTorch 2.0 announcement and conference, they have also posted a comprehensive introduction and technical overview within the Get Started menu at https://pytorch.org/get-started/pytorch-2.0.

This release is intended to improve performance speed as well as add support for Dynamic Shapes and Distributed while still maintaining the same eager-mode development and user experience.

They claim PyTorch’s biggest strength beyond their “amazing community” is that they continue as a first-class Python integration, imperative style, simplicity of the API and options.

PyTorch 2.0 Updates

---

PyTorch 2.0 introduces torch.compile, a compiled mode that accelerates your model without needing to change your model code. On 163 open-source models ranging across vision, NLP, and others, we found that using 2.0 speeds up training by 38-76%.

PyTorch 2.0 is *100%* backward-compatible.
The codebase is the same, the APIs are the same, and the way you write models is the same.
We are calling it 2.0 because it marks adding a significant new set of features.

New components in PyTorch 2.0:

- TorchDynamo generates FX Graphs from bytecode analysis
- AOTAutograd generates backward graphs ahead-of-time
- PrimTorch introduces a small operator set to make backends easier 
- TorchInductor: a DL Compiler powered by OpenAI Triton

---

Sebastian Raschka 📚 @rasbt

PyTorch 2.0 was just announced at the @PyTorch Conference! The focus is speed and accessibility, and the great news is that there are no breaking changes. PyTorch 2.0 will be fully backward compatible. Here's my tl;dw 1/6

PyTorch @PyTorch

Get ready for @soumithchintala to kickoff #PyTorchConference with the keynote, starting soon. Watch the livestream on YouTube: https://t.co/tIbwL34mak After the keynote, tune in for technical talks from PyTorch Engineers. https://t.co/A1obVAHagC

PyTorch 2.0 also introduces `torch.compile`, a new capability that improves PyTorch performance and starts the move for parts of PyTorch from C++ back into Python.

Additionally, several new technologies are included in the update, including: 

• TorchDynamo, which works to safely capture PyTorch programs using Python Frame Evaluation Hooks. 

• AOTAutograd to overload PyTorch’s autograd engine as a tracing autodiff for generating ahead-of-time backward traces.

• PrimTorch to canonicalize ~2000+ PyTorch operators down to ~250 primitive operators that developers can use to build a complete PyTorch backend. 

• TorchInductor is a deep learning compiler that generates code for several accelerators and backends. 

Hugging Face also got in on it.

Tri Dao @tri_dao

We're releasing an optimized implementation of GPT2/GPT3 with FlashAttention🚀! This trains 3-5x faster than the Huggingface version, reaching up to 189 TFLOPs/sec per A100, 60.6% (model) FLOPs util of the theoretical maximum. 1/6 github.com/HazyResearch/f…

github.comflash-attention/training at main · HazyResearch/flash-attentionFast and memory-efficient exact attention. Contribute to HazyResearch/flash-attention development by creating an account on GitHub.

PyTorch Conference Live-Stream Watch

• The most recent generally available release of PyTorch is version 1.13, which came out at the end of October, 2022.

• Chintala emphasized that now is the right time for PyTorch 2.0 because the project is introducing an additional new paradigm in the PyTorch user experience, called torch.compile, that brings solid speedups to users that weren’t possible in the default eager mode of PyTorch 1.0.

• He explained that on about 160 open-source models on which the PyTorch project validated early builds of 2.0, there has been a 43% speedup and they worked reliably with the one-line addition to the codebase. 

It’s vaguely interesting in that:

“Data scientists will be able to do with PyTorch 2.x the same things that they did with 1.x, but they can do them faster and at a larger scale,” Chintala said.

What do you guys think?

As you know, PyTorch gets the first part of its name (Py) from the open-source Python programming language that is widely used in data science. It’s becoming one of the dominant tools in Machine Learning.

PyTorch is a machine learning framework based on the Torch library, used for applications such as computer vision and natural language processing, originally developed by Meta AI and now part of the Linux Foundation umbrella. It is free and open-source software released under the modified BSD license.

“PyTorch 2.0 embodies the future of deep learning frameworks”

Work on Python 2.0 will be ongoing for the next several months with general availability not expected until March 2023.

Thanks for reading!

Get get access to all articles don’t forget to subscribe.