neuralbeans t1_j7jdiqz wrote on February 7, 2023 at 5:18 AM

Reply to comment by beautyofdeduction in Why does my Transformer blow GPU memory? by beautyofdeduction

A sequence length of 6250 is massive! It's not just 6250*6250 since you're not multiplying one float per pair of sequence items. You're multiplying the key and value vectors together per pair of sequence items, and this is done for every attention head (in parallel). I think you're seriously under estimating the problem.

What transformer is this which accepts a sequence length of 6250?

beautyofdeduction OP t1_j7jqohn wrote on February 7, 2023 at 7:56 AM

I wish I can send you my Github. But the original Attention is All You Need paper trained on sequences of length 25000 on multiple K80's (stated by the authors), which has only 12GB vram. Yes they used multiple GPUs, but afaik each GPU needs to be able to handle its own batch. Or maybe not? Again I wish I could show you my code.