Thankfully I'm doing niche enough projects I still get to. Last one was a multi-modal iwae for imputing missing data.

When needed, I usually take an existing architecture and only adapt small parts of it to solve my task. I also wrote a custom autoencoder layer by layer for audio spectrograms (I didn't find an existing model which could do it with my constraints), and a model to convert embeddings from one self-supervised model to another self-supervised model (it's not a complex architecture) while the three models train simultaneously.

Tbh I would prefer to use existing architectures because re-designing an architecture is long to do, to optimize, and to train, but existing models are often very adapted to one task and perform bad on unexpected new tasks. Also you may have constraints (like real-time, memory efficiency etc.) which are not taken into account in easy-to-reuse published models.

Images have pretrained CNNs, but if you want a model to perform self-supervised continual learning and real-time inference on images with just one RTX, it can be harder to find an existing optimized solution for this task.

I’ve never used a pre trained network for anything but my projects are applied to astrophysics which has some very different requirements than NLP or image classification

i was implementing something from a paper that didn't have a public implementation and I wanted to play with it.

It's all frameworks now, some better than others. I haven't written one outside of demos or interviews in years. With that being said, I've modified neural networks a whole bunch. Usually you can just tweak parameters in a config file but sometimes you want additional outputs or to fundamentally change the model in some way...usually minor tweaks codewise.

I'll regularly write custom components, but pretty rarely write whole custom networks. Writing custom prediction heads that capture the task more specifically can improve training efficiency and performance (e.g. doing hierarchical classification where it makes sense, customized suppression postprocessing based on domain knowledge, etc.).

When I do write networks from scratch, they're usually variations on existing architectures anyway. E.g. implementing a 1D or 3D CNN using the same approach as existing 2D CNNs like ResNet or ConvNext. I usually find I'm doing this when I'm in a domain task and don't already have access to pretrained networks that are likely to be reasonable initialisation.

I write custom networks and modules all the time in my research. I imagine most of the researchers in my group do as well.

It was just yesterday. Not a custom neural net. It's just taking different neural networks, arranging them in particular orders, and training them.

The last time I coded a neural net from scratch was some 10 years ago when I coded a Genetic Algorithm and Backpropagation Neural Network. Suffice it to say, the AI field has come a long way since.

The comments in this thread would leave you thinking hardly anyone uses tensorflow, scikit or pytorch any more, but TF alone averages @ around 500k downloads a day.

First I read the title, then I thought "oh they mean like no outside libraries. Just write a neural net from scratch just using math"

Read the comments and was surprised by the answers.

Went back to the body of your post... oh no

Ha! Never

2019. I took a bet that all the training and architecture will be subsumed into some centralized company, where you only really have to worry about dataset.

So in a way it paid off. Now I do everything with hugging face transformers and only worry about dataset haha.

Earlier this year wrote a custom U-Net inspired model for semantic image segmentation in PyTorch. This one I designed to be fully modular so that it's really simple to change the architecture from a config file. Writing the code eloquently took some time, I like the idea of generating blocks of layers to keep the model neat rather than layer by layer (e.g. block includes conv layer, bn, activation, etc.). Probably going to try and do a GAN soon

Lightweight vision module for simple classification for autonomous cars. Have all layers and their sizes memorized at this point lol.

grad school

Can't sell pretrained models, can't sell models who were trained on public data, can't run modern models on edge devices that the clients want because they want it as cheap as possible.

I think the field has gotten mature enough that there's a widening difference between 'using ML as a tool' and 'researching ML'.

As a perhaps poor analogy, it's the difference between an ecologist using an R package to just get some standard ordinations done vs. a statistician developing and writing up a new ordination method.

Not since learning the theory.

There are smarter people than me who can write faster/better code. I’m happy to leverage tool like PyTorch to build and customize nets (like changing the embedding layer in a transformer or tweaking some parts of a CNN) but no need to reinvent the wheel

Not for a while but I started doing it again, did this little thing for Alameda Research and it went pretty great for a while but ... yeah, maybe I shouldn't. (sarcasm, just being facetious with the latest crypto scandal)

Regularly. Perks of using my own framework :)

Well, daily.

But I do research.

While you can solve a lot of problems with out-of-the-box models and a bit of fine-tuning, solving problems in a new way often requires custom/new models.

Not for a few years now— back in the day, one did not simply call MLPClassifier().

I designed a custom architecture to model an analog signal processor with lots of different settings combinations. It was a custom MGU (minimal gated unit) that modulates HiPPO memory according to settings embeddings. Can train in parallel, so much faster than, say, a PyTorch GRU.

Another recent design combines convolution and transformers to model spinal CT scans, which is challenging because a single scan can have a shape like (512, 1, 1024, 1024) that is too large to train for dense tasks like segmentation. If you simply resize to a constant shape, then you lose or distort the physical information embedded in the scans. You don't want a scan of the neck to be the same size as a scan of the whole spine, for instance. So you've got to be more clever than that, and something this specialized doesn't come ready to go out of the box.

Just for learning purposes i wrote code from scratch in python to implement a neural network structure with backpropagation, SGD, with momentum and regularization. It works pretty well. Was able to fit MNIST at 95% on a test set using fully connected architecture.

about a week ago. GCNs lesgoooo

[deleted]

Last week. LSTM to consume both tabular data and a number of text fields. We have tabular transformers and text transformers, but this was easier than tying them both together and trying to make it work.

Almost never, most of the stuff is pre-done and our AI guys usually work on a design and prototype to give to software engineers to write for the company.

Luckily, I’m on that side. We usually grab their model, params and test data and mesh it with our SW to meet the need

Depending on the underlying data its often not allowed when it comes to commercial use (atleast in germany). So I often use custom LSTMs and stuff.

Only when i feel like playing around. Most of the time, fine-tuning OSs models is best effort/value prop

Mostly custom heads and custom loss functions, more than being concerned with the base model. Implementing theatter is increasingly like implementing your own sort function or something: a learning exercise.

I'm also in NLP and I usually just use a pretrained model from HuggingFace as a backbone and build on top of that. It's not usually anything complicated though, maybe an MLP.

You say it's enough to import a pretrained transformer from HuggingFace. I say not even that, you don't need to create a dataset and train a model in most cases, just try a few prompts on GPT-3.

In the last 4 years I worked on an information extraction task, created in-house dataset, and surprise - it seems GPT-3 can solve the task without any fine-tuning. GPT-3 is eating the regular ML engineer and labeller work. What's left to do, just templating prompts in and parsing text out?

Admittedly I'm quite new to ML, but for my master's thesis I wrote several custom CNNs to try and improve on the models of other researchers in the field.

It's actually super rare to deal with custom NN if you have direct customer products. Usually what you see come out of academia is not viable /scalable for any large customer base. You'd be surprised how many baselines power the systems we consider SOTA. Especially in the recommendation systems domain.

A couple of years ago. I was interested in the classification problem for stock price movements. The goal was to predict if the stock yielded positive returns in the future 25-30 days, using daily data of the type provided by Yahoo Finance. I did some feature engineering to derive classical indicators, their moving averages over different time periods and certain normalizations of them so as to have features ranging between 0 and 1. I experimented with various thresholds x and discovered that I get better predictive power by labelling vectors by 1 if the stock returns is at least x %, for some certain x close to 1, than by simply choosing x=0, which means looking only at the direction of the price movement. One drawback, however, was that there was not a clear correlation between the profits and the accuracy of the model: a false positive of, say x/2 %, obviously affected the accuracy in a negative way while it at the same time contributed postively to the profit. Moreover, not defing a recommendation to be a prediction of at least 0.5, but rather something between 0.6 and 0.7 (depending on, for instance the stock index), significantly reduced the number of false positives with negative price movements.

I would still be interested in the question of finding suitable metrics, other than the accuracy, for measuring the performance of the classification algorithm.

Work in NLP. Ever since HuggingFace became mainstream we almost never had to do this.

We used to have to implement the cutting edge stuff ourselves because papers do not come with code, or give code that requires huge amount of work to run. Now they often appear on HF within a few weeks of publication.

The only occasion in the last 2 or 3 years where I wrote a DNN from scratch was when I had to give a short lecture, for pedagogical reasons.

Depending on what you mean by "custom", I still put those things together like legos and fine-tune

Also, I do mostly RL things and a lot of that stuff doesn't have good pretrained things (at least not for my purposes).

I write custom models all the time in order to run them on low resource hardware. It definitely consumes more time but it is worth it.

[deleted]