Submitted by joshadel t3_zaqbwr in MachineLearning
PyTorch 2.0 was just announced at the PyTorch Conference:
https://pytorch.org/get-started/pytorch-2.0/
See also the accompanying twitter thread: https://twitter.com/PyTorch/status/1598708792598069249
God forbid a deep learning framework would not be backwards compatible right lol
TensorFlow flashbacks.... 😵
Pytorch 2 lacking backwards compatibility is the best advertisement there is for using JAX
Is it really, though? Even every minor version increment 1.x introduces backward incompatible changes: https://github.com/pytorch/pytorch/releases
>We introduce a simple function torch.compile that wraps your model and returns a compiled model.
This will be interesting to try out and see how it develops.
I've been using JAX recently and the compiler has kicked my ass in so many ways, it is very hard to get used to and there are many things it just straight up prevents you from doing
Will be interesting to see if PyTorch can make a more enjoyable experience on this front
Imho you need functional programming or undefined behaviour (like in C/C++) to get high optimized code. Undefined behaviour is more pain than functional programming, so i doubt it.
Edit: And even C/C++ compilers like gcc have tags for pure functions to improve optimizations.
It’s not just functional programming, but for instance, you have to use jax.numpy instead of numpy when compiling, but also not every function from numpy is implemented in jax.numpy, and other issues like that
I only have the informations of your link. So I dont know about the other issues you talk about.
But if you set for the functional paradigm it is obvious that you need some jax.numpy and that jax.numpy can not implement every numpy function. Numpy and some of its functions (like inplace updates) are inherent non functional. I cant imagine an other way to fix this.
> Imho you need functional programming or undefined behaviour (like in C/C++) to get high optimized code.
That's not true, see rust.
I am no rust expert therefore convince me that I am wrong, but that is only true if you dont use unsafe blocks. This would exclude using CUDA and as far as I know in some cases you need unsafe blocks to get C like performance.
But even if I am wrong and no undefined behaviour is needed. Even Rust has a pure function attribute to improve optimizations.
It just makes sense to use this improvements in libraries like pytorch/jax. Especially since mainly mathematical operations are performed that are pure functions anyway.
I'm no expert either, but you're right that using CUDA requires use of unsafe. I believe kernels are even written in C through macros.
However, using unsafe does not necessarily mean UB. You preferably want to avoid that regardless. And UB is not the only way a compiler can optimize. Unsafe code simply means that you are responsible for memory safety, not that it should be ignored.
I don't know, you're talking about UB as if it was a feature and not an unfortunate development of compilers over the years.
In fact, Rust made it very clear that if you rely on UB that's your pain. Don't come crying in a week when your shit does not compile anymore. No guarantees are made, and no extra consideration is made to maintain compatibility with programs that make up their own rules.
>However, using unsafe does not necessarily mean UB. You preferably want to avoid that regardless.
>Unsafe code simply means that you are responsible for memory safety, not that it should be ignored.
Maybe I am wrong but I think you misunderstand UB. Of course you want to avoid UB and have memory safety in your code/executable because otherwise you can not argue about the program anymore. But you want UB (at least in C/C++ the language I work with) in your standard. UB is more like a promise of the programmer to not do specific things. The compiler assumes the code contains no UB and optimizies like that. See for example signed integer overflow. Because the compiler knows this is UB and the programmer promised to not allow it he can use better optimizations. Rust does not have this "feature" in safe blocks and produces less optimal code.
>And UB is not the only way a compiler can optimize.
I would not disagree about that. But if you want the last .x% of performance increase than you need it too. Especially if you want your language to work on different systems. Because even Hardware can have UB.
The only other option (as far as I know) you have to get some comperable (with UB assumption) performance is to rely on other assumptions like functions have no side effects etc.
>I don't know, you're talking about UB as if it was a feature and not an unfortunate development of compilers over the years.
As language specification it is like a feature. In the binary it is a bug. I have read enough discussions of UB in C++ threads to know that a lot of C++ developers dont see UB as unfortunate development of compilers.
>In fact, Rust made it very clear that if you rely on UB that's your pain.
By the way this is the sentence why I think you that you misunderstand UB. As mentioned: You should never rely on UB you promised the compiler to avoid it. And by avoiding it the compiler can work better.
Will this will be like TF I wonder.
Previews of PyTorch 2.0. i.e. you can get from the nightly builds.
> We expect to ship the first stable 2.0 release in early March 2023.
The speed up is only available for newer Volta and Ampere GPUs for more. Hopefully with primTorch it’s easier to port to other accelerators in the long run. And the speed up is less prominent for consumer GPUs.
Wow, this sounds pretty exciting. I wonder how the speed will compare to JAX or Julia.
One of my first thoughts as well! Is there any reason PT's speed ceiling would be lower than JAX's? Ik PyTorch-XLA is a thing but not sure about its current status
These are some exciting sets of features!! It's especially great that there are no breaking changes.
I personally like semver a lot, so the only thing I don't like about this announcement is that they bumped the major version to 2.0 even though there is full backwards compatibility.
SemVer allows for significant internal or additive changes to cause a major revision, so I wouldn't worry about it.
It allows for breaking changes; it doesn't require it.
Or I'm sure they could do another 1.x release with a new "hello world" function that they change the signature on for the 2.0 version otherwise.
Someone needed to be promoted to distinguished
Nothing about edge hardware support (their functions are in beta for quite some time now)
This is what the export feature is about but it's still in early days https://pytorch.org/get-started/pytorch-2.0/#inference-and-export
Why going towards JAX ONLY. As a quick survey: am I the only one who wants a high level differentiable framework in a strongly typed language?
All I want for Christmas is a strongly typed, low level, non-garbage-collected, safe programming language with pythonic syntax and first class tensor types and native compile-time autodiff. Is that too much to ask for?
You can always build one with lisp /s
Lol, kind of the opposite -- I am not a fan of python syntax.
The rest? All except for low-level and non0garbage collected.
I mean... in case Santa were listening...
Have you tried Dex? https://github.com/google-research/dex-lang It is in a relatively early stage, but it is exploring some interesting parts of the design space.
I did, yes, but I found the syntax counterintuitive. It is very python-like, but its syntax was conceived to not include types declarations in the first place, and only later adapted to do so.
When I say high level differentiable framework in a strongly typed language I imagine to take something that works already as stronlgy typed, and then adapt it to automatic differentiation and jit compilation -- but not the opposite.
I refer to an hypothetical language that is, for example, what C# is to the C++. Similar syntax, higher level.
Does that make sense?
Flux in Julia is your answer. Although Julia isn't technically strongly typed (it's optionally typed for multiple dispatch) it's about as good as it gets imo
Anyone personally tested the speedups? Please share if you did
long live pytorch
so, with torch.compile people can keep writing graph-unfriendly code with random dynamic shapes and direct python code over tensors ?
that's the goal yes, although dynamic shape support is still under works
This is pretty cool, hoping they can give specific by-gpu benchmarks at some point.
I want a compile to shader (glsl) feature next!
The new compiler is so cool!!
Though virtually no speed-up on ViT: https://pbs.twimg.com/media/Fi_CUQRWQAAL-rf?format=png&name=large. Anyone has an idea on why?
fixed-size sequences?
That's a good point. Though it's still unclear to me why that would result in no speedup.
What dies it mean for my MNIST model?
I like how flexible they are about different compilation approaches. In TF2 the problem was that you always need to wrap everything in tf.function to get the performance improvements. Debugging it was a nightmare since for more complicated pipelines it could take several minutes just to compile the graph.
Will this allow to finally JIT-compile custom backward? (in python)
Normally I build my model with TF, so I don't have a deep understanding of PyTorch, so I don't understand why this .compile thing is important. Can someone explain to me?
Basically, pytorch is like writing tensorflow 2 eager execution code. Now with compile maybe they create a static computational graph like tensorflow 1.x or tf.function?
mileseverett t1_iyn1es7 wrote
100% backwards compatibility. Thank god.