> I know it’s common for massive projects to use Fortran in order to train NN

Is it? I'm not aware of any high profile / large scale ml projects recently written in it. My understanding is they mostly use python for the model development, and then C/C++ for the actual math. Fwiw i think parts of numpy are written in fortran though.

Many neural networks that are used in like physics chemistry and other fields are often written in Fortran and then trained on supercomputers. It’s mainly because Fortran is really easy to parallelize and even has built-in support for linear algebra. additionally most supercomputers run on Linux and to my knowledge fortran runs really well on Linux OS. I am also planning on learning C/C++ down the road for similar applications but currently I just think Fortran is more fun lol and again I’m doing it just for the fun of it.

Edit: this is to my knowledge again I am EXTREMELY new to Fortran but atleast from my google research it seems to be used in a range of large scale projects.

Sounds like these were written back in the days when GPUs didn’t exist. I believe a consumer GPU these days can beat any supercomputer of the past

If your referring to Fortran, there’s gpu support using CUDA Fortran, the purpose of using computer clusters and CUDA Fortran and all that stuff seems to only be for ridiculously large data sets. To my knowledge NASA used such ideas when making simulations for rover landings and other things of the sort.

Fortran is pretty much a dead language though. People still use Fortran in their DFT computation only because no one has ported them over to a modern language. Since you are picking up a new language, just pick up Python to get the required support or you will be having a lot of trouble finding help.

I would say I already have a good understanding on Python and have done many Deep learning and machine learning projects in the past using tensorflow and even making my own crappy framework. I’m more just looking towards learning more about the bare bone fundamentals of how to parallelize neural networks by making them from scratch so I can best understand it all.

I'm not going to jump the "Fortran is dead" bandwagon as it is a language that continues to get used in simulation code and I feel it's never really going to be replaced. As you mentioned it has good properties for writing that kind of code, good compilers, etc.

That said, it is a very niche language, used mostly in the physics simulation & supercomputing world. And idk how popular NN are in that sector, but it seems to me it's a niche inside a niche... So you will need to dig a bit to get answers. Maybe the people at r/Fortran will know more.

There is a lot of outdated code in physics and chemistry. It is in Fortran because no one wants to rewrite it.

It is actually more difficult to parallelize fortran code on supercomputers compared to modern distributed computing libraries like HPX, which is a C++ library. Fortran will also perform much worse as you won't beat the DAG schedulers in HPX.

Fortran is also slower, because you can't write generic high performance code. This is also true of C. You need something akin to templating which allows you to optimize expressions at compile-time like you can do in C++. You need generic code for all but the most trivial operations, else it is very difficult and time consuming to build more complex operations like those that build into neural networks.

Additionally, CUDA is native C++ (and again can be optimized generically which you can't do once you have C or Fortran APIs), so if you want to seriously take advantage of vectorization, you should look at using GPUs. It is likely that if a place has a supercomputer, they probably also have a GPU cluster.

All these reasons are why C++ is the king of performance programming.

I would look at C++ frameworks. Never heard of any sane implementation in pure fortran.

I have a book on Fortran dated somewhere in the seventies. Thats also the last i heard of it.

Hey, do what works for you, kid. (I say that respectfully). If you're passionate about writing a neural network in Fortran...

...then do it. If you follow one of the practical solutions here and it makes you miserable, then don't do that. There are easier ways than Fortran, but if you accept that it's not necessarily the most efficient or easy, like, at all, way to go about things, and this is a passion project for fun, then that's something you can learn with. Whatever fills your sails and gets you _learning things_.

I think you want to ask yourself why Fortran specifically (it's a low level language, you like it in particular, older supercomputing history, etc?)

Once you have written down all of the _why_ for what you want to do, make sure that you've chosen the right building blocks (the _what_) for it.

I hope this helps you on your journey. Much love and care from this fair enby over here! <3 <3 <3 :DDDD

Well, until recently, NumPy apparently was made of Fortran! :OOOO

That’s a very good idea. I probably will do that, after getting a lot of these responses I’ve been thinking about just doing C++

never too old to learn, i didnt know that. Amazing.

Nothing about Cobol?

I liked Forth, but that died.

>I know it’s common for massive projects to use Fortran in order to train NN.

It is definitely not common. Yes, Fortran is used in scientific computation applications due to efficient and well tested linear algebra libraries and other numerical computing legacy code.

Fortran code is or can be used under-the-hood of higher level libraries / languages, such as Numpy for Python or Matlab. Even PyTorch uses LAPACK for linear algebra computations when running on the CPU. In this sense, yes, Fortran code is used, indirectly for training NNs. But using Fortran to actually implement a NN model and train it is virtually unheard of, as far as I know.

Maybe having a look at LAPACK will give you more insight.

I did an applied Math masters in 2016 and they teached us Fortran (Matlab too), along with the usual commercial software such as ansys + ofc all the PDE theory necessary.

Point being: it's niche but it's still there. Classmates who ended up tightly adhering to the masters career path now write Fortran for a living.

And don't try to sell me "but C++ does the same and it's better/more modern". I've written Fortran, I've written C++, and Fortran is neither arcane nor hard, especially when you use it for its intended purpose (FORmula TRANslation, i.e. physics sims), in fact it blows C++ out of the water in usability if you are not a computer scientist... Which is why physicists and mathematicians keep using it.

Wont sell anything I have no hard opinion on it, where i worked fortran, cobol just disappeared.

I never used it being in real time machine language (assembler) programming, which is another universe.

I encourage you to continue your own exploration, regardless of what anyone says "what everybody else is doing". The truth is, all of this is applied math, and the computers are merely where we do our work. Personally, I find this field much more interesting at the algorithm level: If you do something genuinely interesting, it is not somehow less valid because of the tools you used.

Maybe it is just that physicists love Fortran. Probably for good reasons and that do not have something with deep learning. Sometimes you stick to what you really know well. I have a math background and I was really surprised when a physicist friend said he coded in Fortran...

Parts of scipy is in FORTRAN, not numpy afaik!

Chris Sawyer wrote Rollercoaster Tycoon purely in assembly!

I still like assembler, whatever platform. It gives you a very direct feel between machine and function. It is more work though.