Does it though? There was a reproducibility survey recently that found that many optimizers claiming better performance did not in fact work for anything other than the tasks tested in their papers.

Essentially they were doing hyperparameter tuning - just the hyperparameter was the optimizer design itself.

I've just skimmed the paper, but this is a confusing result. I can see a simpler optimizer paying off when using similar amounts of computing due to being able to run more iterations, but they claim it's also better on a per-iteration basis across the entire learning task. There's not a lot going on in this algorithm, so where is the magic coming from?

It's kind of hard to believe that while people were experimenting with all these more complex optimizers no one tried something this simple and saw that it had state-of-the-art results.

I'm waiting for deberta glue/superglue results, it's weird that they picked T5 for that

Better than AdamW if (a) the model is a transformer, (b) not a lot of augmentations are used. Otherwise, the improvements are not that large. I doubt this optimizer works well with regular CNNs like efficientnet or convnext.

Do learned optimizer people seriously believe this is the direction we should be going?

This is a hand-designed optimizer. By definition, learned optimizer researchers would rather we learn an optimizer than hand-design one.

Learned optimizers are probably the future, but the compute budget required to create one is prohibitive.

ML acronyms are getting out of hand, just use any letter from any of the words I guess...

"It is more memory-efficient than Adam as it only keeps track of the momentum."

While this is technically true, is this a joke?

u/ExponentialCookie - In the Code Implementation link, lucidrains writes about reproducibility issues and tuning, both issues brought up in these comments.

> ...just the hyperparameter was the optimizer design itself.

Probably one of the best things I've read today lol. Reminds me of when old colleagues of mine would have lists of different PyTorch optimizers and just loop through them.

Care to elaborate?

I would have named it "Eve", as she came after Adam (if you are into these stories)

That would have been a cool name!

Wait that's so much better

They might be thinking in a different direction than me, but the majority of Memory use during training is not from the model weights or optimizer state in most cases. It comes from tracking all the activations of the training batch. If you think about a cnn, each filter gets used across the whole image so you will have many more activations than filters. So optimizer memory savings has very limited benefits

[deleted]

those are way less. for every vector of activations you usually have that squared in weights time 2 or 3 depending of how many momentum values you keep.

EVolved sign momEntum (EVE) 🤣

Not true, in any case with convolution, attention, or recurrence, which are most modern applications. In all of these cases the activation count grows with how often weights are reused as well as with batch size. Those dominate optimizer memory usage unless you used a tiny batch size.

That's why checkpointing can be useful. This paper does a solid job covering memory usage: https://scholar.google.com/scholar?q=low+memory+neural+network+training+checkpoint&hl=en&as_sdt=0&as_vis=1&oi=scholart#d=gs_qabs&t=1676575377350&u=%23p%3DOLSwmmdygaoJ

yea it depends. Even just batch-size makes a difference. But for really big models, I'd assume that the number of weights far outweighs the number of activations.

Yeah very configuration dependent, but larger batch sizes usually learn faster so there's a tendency to lean into that

That one is already in use :P

https://arxiv.org/abs/1611.01505

I think learned optimizers have potential but this is disappointing. Nothing revolutionary in there…there are already sign based optimizers and this is just a slightly different take. I see learned optimizers as the possibility of getting unintuitive results but this just could have been thrown together by some grad student. Random but not surprising.

so, technically this is binary optimizer that updates the weight to either -1 or +1 multiplied by lr. Should be tested with "Learning Rate Dropout", i.e. 30% chance to update with -1/+1, otherwise no update.

Ever heard of large models?

I really think we need an intermediate between conference papers and arxiv, to just evaluate how reproducible/sane the paper is without evaluating whether it is important or not.

Because at this stage I genuinely can't tell if that's a press release, a report in a paper form, or an actual paper.

In my limited testing on a UNet like CNN, it doesnt even come close to the performance of adam sadly. With that said, i might be doing something wrong.