I did some reasearch a few years ago into timeseries forecasting, specifically day-ahead forecasting of photovoltaics from historical data of frequency x (15min) and general weatherforecasts (freq 1h) and we did notice that attention made our LSTM S2S model jump past the (then) state of the art. We published a paper and then I started looking into transformers instead of the LSTM based S2S model and they did perform better albeit this never made it to a paper due to other circumstances.

I think now with the better understanding of transformers we have I would excpect the results to be even clearer, assuming sufficient data and the right setup.

think is a lot of forecasting tasks have low datamass and the feature distilling nature of a transformer might not be the best choice. and then tranining transformers is still a little tricky for any non-vanilla application. Floating point rgression is somewhat different than a multi-label type output after all