How To Scale Transformers’ Memory up to 262K Tokens With a Minor Change?

Submitted by rezayazdanfar t3_11qfl2o in deeplearning

I just published my latest medium article.

This article is a fabulous attempt to leverage language models in memorizing information by transformers with the least required effort by inserting only an external memory near the last layer of the transformer.

We can use this to retrieve information that the transformer is trained with which helps the reliability of predictions.

Feel free to share this and/or contact me directly.

https://medium.com/towards-artificial-intelligence/extending-transformers-by-memorizing-up-to-262k-tokens-f9e066108777

15

Comments

You must log in or register to comment.

PM_ME_ENFP_MEMES t1_jc5yjdv wrote

Replying to rezayazdanfar (#2,226,012)

Own it bro, don’t listen to the haters. If your work is fabulous, call it fabulous, literally nobody can stop you and nobody that appreciates your work will care what you call it as long as it contributes to moving things forward.

1

WallyMetropolis t1_jc6qi9w wrote

Replying to rezayazdanfar (#2,226,012)

I see. Typically, when you say "this thing" you're referring to the most recent mention of that thing. So "Here's my article. This article is fabulous." is probably not the structure you want. You really should reference the paper or the researchers you're writing about straight away, if that's what you're doing here. Even after skimming your article, that isn't clear.

It's a cautionary tale about "this" really. A tip that has helped my writing is to accumulate a list of words that I tend to over-use that add nothing and search for them in the editing process. Words like "this, really, just, again," and so on.

2

DeepLearningStudent t1_jc7ly6x wrote

Holy crap, if this works that’s a huge game changer for a problem I’m working on.

2