This is quite intriguing. It ranks in the top 10 percent of participants who played more than one game, which at first glance isn't as impressive as AlphaZero, for example. Then however, you remember that diplomacy requires players to actively converse & cooperate with other players, which sets many variables out of the AI's sphere of influence. I'm not too familiar with Diplomacy, but I imagine the top 1% of players who played more than one game have played only 4-5, and got lucky by winning them all. Feel free to prove me wrong on that, though.

Chances are that this is limited by its language model, a field in which we see rapid improvements every day.

Edit - just want to add that I've been increasingly fond of Meta AI recently. Phenomenal stuff coming from them, and they're open sourcing CICERO.