matrix multiplications require doing additions (and subtractions) and multiplications.

GPUs can do additions (and subtractions) faster than multiplications.

by rejiggering the way the matrix multiplication is written you can use less multiplications and more additions thus it runs faster on the same hardware.

https://en.wikipedia.org/wiki/Strassen_algorithm

>Volker Strassen first published this algorithm in 1969

.....

>In late-2022, AlphaTensor was able to construct a faster algorithm for multiplying matrices for small sizes (e.g. specifically over the field Z 2 \mathbb {Z} _{2} 4x4 matrices in 47 multiplications versus 49 by the Strassen algorithm, or 64 using the naive algorithm).[2] AlphaTensor's results of 96 multiplications for 5x5 matrices over any field (compared to 98 by the Strassen algorithm) was reduced to 95 a week later with further human optimization.