Submitted by Chemont t3_109z8om in MachineLearning
I recently came across " Confident Adaptive Language Modeling " which allows Transformers to exit early during inference and not use all model layers if a token is easy to predict. Is there any research on basically doing the opposite and allowing Transformers to spent more compute on tokens that are very hard to predict?
I should have been clearer with my question. What I was wondering was, if there are any extensions to the Transformer architecture that allow it to, in theory, spent indefinite amounts of compute on one token. I suppose one could train a very deep Transformer, use CALM during inference and only use all of the layers for tokens which are difficult to predict, but this would still arbitrarily limit the maximum amount of compute per token.
You'll still pay the full price at train time, right? Early decoding works by using decoders on earlier levels at train time. Conversely, if you want to spend more on some tokens, at train time, you will need to have more layers, so at some point you will hit your memory/complexity limits.
No but if you use a decoder model (autoregressive) you can generate more tokens for the same task, depending on its difficulty. Chain-of-thought makes use of this trick.
Yes. This thread made me think of Universal Transformers which has dynamic halting and has been around for a while now: https://openreview.net/forum?id=HyzdRiR9Y7
Did this ever get any traction?
That depends on what you mean. I don't think any of the LLMs use it, but it has some citations and follow-up literature.
[deleted]
Chain-of-thought prompting does this for the LM transformers. It can complete the harder objectives by using more computations.
I know it doesn't solve the general case, but you may take inspiration from it in other domains.
Universal Transformer: https://arxiv.org/abs/1807.03819
Ponder Net: https://arxiv.org/abs/2107.05407
Deep Equilibrium Net: https://arxiv.org/abs/1909.01377
http://www.gatsby.ucl.ac.uk/~balaji/udl2021/accepted-papers/UDL2021-paper-072.pdf
There's albert which reuses the same layers throughout, I can see a case where albert is used, and a decoder thats a few neuros is also present, where at each step it will use a token in the input to determine if its time to stop, similarly reso net did this for Q&A
What does early stopping inference mean tho
At train time, you plug decoders at many levels with the same objective, you can find out if some things can be decoded earlier, using an additional network that outputs a sort of confidence. At inference time, you run the layers one by one, and stop when the confidence is high. which allows you to skip some computations. (It's probably a simplistic description, feel free to correct me)
[removed]
FWIW in certain sense this goes against the design philosophy of transformers, which is to jointly compute all representations within a layer at once, to maximize the degree of parallelism on GPUs and other accelerators.
How would you even do that? Once you have run inference through all layers, you can not just randomly pull additional layers out of thin air, can you?
amrit_za t1_j418a4l wrote
It sounds like what you're considering the "opposite" is just a reframing of original task i.e. if a token is difficult to predict, then more layers (and therefore compute) would be used used. If it's easy, fewer layers. Am I missing something from what you're asking?