PyTorch 2.0 Release
PyTorch 2.0 release accelerates open-source machine learning
Hey Guys,
Just a quick note, Meta is releasing PyTorch 2.0.
PyTorch Important for Machine Learning
As you know, among the most widely used machine learning (ML) technologies today is the open-source PyTorch framework.
It’s becoming pretty dominant. PyTorch got its start at Facebook in 2016 with the 1.0 release debuting in 2018. It’s come pretty far in just five years.
In September 2022, Meta moved the PyTorch project to the new PyTorch Foundation, which is operated by the Linux Foundation. Today, December 2nd, 2022, PyTorch developers took the next major step forward for PyTorch, announcing the first experimental release of PyTorch 2.0.
So it looks like the new release promises to help accelerate ML training and development, while still maintaining backward-compatibility with existing PyTorch application code.
To complement the PyTorch 2.0 announcement and conference, they have also posted a comprehensive introduction and technical overview within the Get Started menu at https://pytorch.org/get-started/pytorch-2.0.
This release is intended to improve performance speed as well as add support for Dynamic Shapes and Distributed while still maintaining the same eager-mode development and user experience.
They claim PyTorch’s biggest strength beyond their “amazing community” is that they continue as a first-class Python integration, imperative style, simplicity of the API and options.
PyTorch 2.0 Updates
PyTorch 2.0 introduces torch.compile, a compiled mode that accelerates your model without needing to change your model code. On 163 open-source models ranging across vision, NLP, and others, we found that using 2.0 speeds up training by 38-76%.
PyTorch 2.0 is *100%* backward-compatible.
The codebase is the same, the APIs are the same, and the way you write models is the same.
We are calling it 2.0 because it marks adding a significant new set of features.
New components in PyTorch 2.0:
- TorchDynamo generates FX Graphs from bytecode analysis
- AOTAutograd generates backward graphs ahead-of-time
- PrimTorch introduces a small operator set to make backends easier
- TorchInductor: a DL Compiler powered by OpenAI Triton
PyTorch 2.0 also introduces `torch.compile`, a new capability that improves PyTorch performance and starts the move for parts of PyTorch from C++ back into Python.
Additionally, several new technologies are included in the update, including:
TorchDynamo, which works to safely capture PyTorch programs using Python Frame Evaluation Hooks.
AOTAutograd to overload PyTorch’s autograd engine as a tracing autodiff for generating ahead-of-time backward traces.
PrimTorch to canonicalize ~2000+ PyTorch operators down to ~250 primitive operators that developers can use to build a complete PyTorch backend.
TorchInductor is a deep learning compiler that generates code for several accelerators and backends.
Hugging Face also got in on it.
PyTorch Conference Live-Stream Watch
The most recent generally available release of PyTorch is version 1.13, which came out at the end of October, 2022.
Chintala emphasized that now is the right time for PyTorch 2.0 because the project is introducing an additional new paradigm in the PyTorch user experience, called torch.compile, that brings solid speedups to users that weren’t possible in the default eager mode of PyTorch 1.0.
He explained that on about 160 open-source models on which the PyTorch project validated early builds of 2.0, there has been a 43% speedup and they worked reliably with the one-line addition to the codebase.
It’s vaguely interesting in that:
“Data scientists will be able to do with PyTorch 2.x the same things that they did with 1.x, but they can do them faster and at a larger scale,” Chintala said.
What do you guys think?
As you know, PyTorch gets the first part of its name (Py) from the open-source Python programming language that is widely used in data science. It’s becoming one of the dominant tools in Machine Learning.
PyTorch is a machine learning framework based on the Torch library, used for applications such as computer vision and natural language processing, originally developed by Meta AI and now part of the Linux Foundation umbrella. It is free and open-source software released under the modified BSD license.
“PyTorch 2.0 embodies the future of deep learning frameworks”
Work on Python 2.0 will be ongoing for the next several months with general availability not expected until March 2023.
Thanks for reading!
Get get access to all articles don’t forget to subscribe.