Domain specific LLM's need not to be huge like these LLM's like chatgpt. They have world knowledge. In most of the settings, we don't need that.

Let say if for some example we dig into these attention maps, and find some perspective of some head for attending words. For an example in gpt some head focus on parts of speech. Will it always reliably do it for all example? What do you think. Can we manually evaluate and categorize the learnings??

They use parallel processing and threads that's sufficient for ml not so deep

Wow, this is insanely great for factual things. I tried to generate for oops python concepts. The texts were good but the images it picked was not of python codes. But great solution, great start