My answer is less refined than some of the other ones, and my experience with RL is minimal, but wouldn't the action space be too large? Could you contain it to any word/phrase (near infinite space)? You could try limiting it to single letters, but similar to how CNNs work, you'd be missing out on the relationship between letters and you'd still have a 26 character action space, assuming you don't use punctuation or numbers. My friend spent two years working on a RL algorithm with only a 6 action space... I can't imagine 4x that