Very neat! Would love to see a version built with fewer filters (secondary models)--i.e., more grounded in a singular, "base" model // less hand-tweaking--but otherwise very cool. (Although wouldn't surprise me if simply upgrading the model size went a long way here.)

Very cool work. I saw this on my LinkedIn feed and immediately had to share it with my fiancé who is a huge fan of risk and diplomacy. To me, this seems like a much bigger deal than AlphaGo - can someone give me a sanity check?

I’m also interested in how much thought was put into the persuasiveness of generated messages when making a proposal. It seems like something way out of the scope of RL, but still quite important to optimize. I am just… astounded reading over that convo between France and Turkey. If you have time, would you mind offering some insight into the impressive “salesmanship” of CICERO’s language model?

> Example dialogues

ITALY: So, what are you wearing?

Fuck Meta

A strange game. The only winning move is not to play. How about a nice game of chess?

E: -7? It was a movie quote guys...

Meta as a company might be shit, but their AI research is incredible.

Doubt we will get a playable demo/version of this?

AlphaGo was beating the best, this is, according to the post, a top 10% player, which most likely means 9.x% percentile. This also includes players with more than 1 game, but they played 40 games. So just by allowing a bunch of 2 games player they up their stats. A fair comparison would have been to take players with at least 40 games, sample 40 games randomly and compute the score, and then check the performance on this subtrata.

Not to take away anything from the team, but given how the the results are framed, my instinct is to believe that this is a bit oversold.

9.x% percentile? i assume you meant 90+ percentile?

There's no comparison to prior full-press Diplomacy agents, but if I'm reading the prior-work cites right, this is because basically none of them work - not only do they not beat humans, they apparently don't even always improve over themselves playing the game as if it was no-press Diplomacy (ie not using dialogue at all). That gives an idea how big a jump this is for full-press Diplomacy.

Author Adam Lerer on speed of progress:

> In 2019 Noam Brown and I decided to tackle Diplomacy because it was the hardest game for AI we could think of and went beyond moving pieces on a board to cooperating with people through language. We thought human-level play was a decade away.

I assume so. Otherwise that's a bottom of the barrel player...

Nuance (:

iirc FAIR has work playing hannabi, which require some level of (non-verbal) communication. So a lot of the insights can be leveraged here as well.

wrong sub-reddit.

we here read more than 2 words.

ITALY: asl, pls?

UNKNOWN: f, 19, Paris

ITALY: Oh, so you must be french.

Listened to a podcast with Andrej Karpathy recently, and his intuition for the future of LLM is that we'll see more collaboration and stacking of models, sort of a "council of GPT's" kind of approach, where you have models trained on particular tasks working together towards the goal.

Whatever the future holds, I'm betting we'll see constant improvements over the next few years, before we see a new revolutionary one-model take.

Yeah, understood, but that wasn't really what was going on here (unless you take a really expansive definition).

They were basically doing a ton of hand-calibration of a very large # of models, to achieve the desired end-goal performance--if you read the supplementary materials, you'll see that they did a lot of very fiddly work to select model output thresholds, build training data, etc.

On the one hand, I don't want to sound overly critical of a pretty cool end-product.

On the other, it really looks a lot more like a "product", in the same way that any gaming AI would be, than a singular (or close to it) AI system which is learning to play the game.

But they specifically created a model for playing Diplomacy - not a process for building board game playing models. With the right architecture and processes then they could probably do away with most of that hand-calibration stuff but the goal here was to create a model that does one thing.

The code is available so it seems like someone could host one pretty easily. Not sure the system requirements though.

I'm an 8 word man myself

Hmm. Did you read the full paper?

They didn't create a model that does one thing.

They built a whole host of models, with high levels of hand calibration, each configured for a separate task.

I'm one of of the paper authors:

You can see a full anonymized table of scores and ranks near the end of the Supplementary Material file linked for download at the end of the Science article. No player other than Cicero played anywhere close to 40 games, so such a procedure wouldn't be possible. Each game takes hours and requires scheduling 6 players to be simultaneously available, so understandably many players, including many good players, only played a handful of games each. If you restricted to, say, players with >= 5 games, Cicero would be 2/19.

We don't make a claim of being superhuman as AlphaGo did - we believe Cicero in this setting is at the level of a strong human player but not superhuman. We worked with top Diplomacy experts who have given us this feedback.

One thing to keep in mind is that Diplomacy has variance: there is practical luck in which players choose to ally with you or someone else, or whether you guess right or wrong in things like coin-flip tactical situations. So similar to, e.g. poker, even a middling player may occasionally win big in the short-run against top-level players to a degree that would not hold up in the long run. This means including players with too few games can sometimes have the exact opposite bias and make a strong result seem worse by comparison. In that quoted stat, we chose a threshold of > 1 game as a compromise between mitigating the most misleading tail of that bias, while still including as many players as possible rather than picking a higher threshold and arbitrarily cutting out large chunks of the player population from the comparison.

But of course, none of that ultimately matters since you can still check out the full list yourself.

If you're interested in a bit more context on the player pool: the setting was a casual but competitive online blitz Diplomacy league advertised at various times in some of the main online Diplomacy community sites. Many newer players signed up and played, but also experienced players, and as an organized league I'd expect the overall average level of play to be a little higher than, e.g. generic online games.

And thank you and others for raising such questions - it's been fun and interesting to see discussions like this.

Do you happen to remember which podcast that was? Sounds interesting

Lex Fridman's: https://lexfridman.com/andrej-karpathy/

Should be around here, if you want a direct timestamp, although I found the entire podcast really worth while.

(38:18) – Transformers

(46:34) – Language models

(56:45) – Bots

Thank you!

Great results! Some feedback:

• I'm somewhat unsatisfied with the amount of human engineering / annotation pipelines that went into the agent. Importantly the "intention" mechanisms, which seem to be a key part of making the dialogue -> planning part tractable.
• This annoyance somewhat extends to the "message filtering mechanisms" to prevent non-sensical, incoherent messages, as this seems more of a hack. Really, the agent should learn to converse from the objective of being an optimal player (amongst other humans). Because if it starts speaking gibberish, then other human players can tell it is an AI. This would most likely be a bad outcome for the agent (unless the humans are blue-pilled).
• From what I gather, it seems like it is only trained on "truthful" subset of the dialogue data, which means the agent cannot lie. Deceit seems pretty important for winning Diplomacy.
• The sections on planning are not easy to understand concretely, specifically "Dialogue-conditional planning" and "Self-play reinforcement learning for improved value estimation". The authors seem to paraphrase the math and logic in words and omit equations to keep it high level, but this just makes everything more vague. Luckily, the supplemental seems to have the details.
• Thanks for publishing the code. This is very important for the research community. I hope FAIR continues to do this.

Also, the PDF from science.org is terrible. I can't even highlight lines with my Mac's preview app. Please fix that if you get a chance!

It seems like the vast majority of CICERO's pitches are "here's an optimal play for you, you should do it not only because it's good for you but also because it's good for us." In other words, pointing players towards rationality. Of course, high level players in any social game are far more likely than their less skilled counterparts to want to make the rational play, so it's likely that there's some selection bias influencing how effective that salesmanship is. However, even high level players are governed by the emotions that break down game theory!

Here's an example case: I'm curious to see how CICERO responded in situations where they talk to Human A about a plan that requires Human B, but A doesn't trust B. How does CICERO respond to that? It may very well be that it doesn't get in those spots because it thinks about who's likeliest to align with whom and in what way. In this sense, it's playing to its strengths and not attempting plays it can't execute, which is an impressive strategic feat. But of course, I'm interested in seeing it try things it can't do - in this case, try a different mode of persuasion.