Have you mainly been reading light novels, or are you reading other novels as well? It's possible that what your reading just doesn't do visual imagery very well, either because the translator did a poor job or just because it's poorly written.

For example, there is one Japanese light novel that I dropped because I also couldn't really 'see' what was happening, even though I don't normally have a problem with that. It may have been a translation issue, but I suspect the novel itself was just bad. If your looking for good imagery (in English), I'd aim for novel originally written in English. If your looking for something like a light novel, Korean webnovels are usually translated really well (and seem to have imagery that translates well to English). Out of Japanese, Chinese, and Korean web/light novels, Japanese seem to be the most difficult for me to follow.

Of course, maybe the aphantasia is the issue, but I can't really comment on that since I find it difficult to even imagine what that's like.