Bridging rl and planning and connecting both to NLP has been an area of interest for a while. You could adjust large language models to have more of a personality and understanding of the concept of a state. This is at least a good demonstration that those three concepts can be connected well in a HCI setting

While CICERO is only capable of playing Diplomacy, the underlying technology is relevant to many real-world applications. We think others will be able to build on this research in a way that might lead things like better AI personal assistants or NPCs in the metaverse.I think the way we integrated strategic reasoning with NLP was novel and has implications for future research.

We've open-sourced all the models and code. We're also making the training data available to researchers who apply through our RFP. Running the full CICERO agent, including the strategic reasoning component, is quite expensive. The raw models by themselves are more manageable though. -NB

Backstabbing tends to get devalued by CICERO. It has long been my thinking that backstabbing is a poor option in the game and I always feel like I fail when I have to do it, and CICERO seems to agree with me. It gets clearly better results when it is honest and collaborates with allies over the long term. If you forced it to play a pure tactical style game in an environment with communication it would perform poorly, and I think there's a marker there for human players who want to get better as well as some interesting AI ethics ideas that can be explored in future. -AG

See this article about a 3x world champion.

> I asked Goff about any major falsehoods or betrayals that helped him in his victories. He paused to think, then said in his soft-spoken way: “Well, there may have been a few deceptive omissions on my part but, no, I didn’t tell a single outright lie the entire tournament.”

CICERO's dialogue model is trained to generate messages that honestly correspond to the intents (actions for itself and for its dialogue partner) that are inputs to the model, and CICERO always inputs the action it actually intends to take. That said, that doesn't mean CICERO will never attack any particular player. If it chooses to do so, it might strategically withhold details of its plans from that player. -NB

I've talked to a few folks about whether this kind of research is applicable to financial markets and the short answer I've gotten is "not directly". I think there are many more promising directions to take this research, like personal assistants and modeling drivers on roads. -NB

[deleted]

Meta has no plans to turn CICERO into a product and that was never the goal. This is purely AI research that we have open sourced for the wider research community. I think there are a lot of valuable lessons that the research community can learn from this project. -NB

I started grad school in 2012 and technically defended in 2020, but I actually left the PhD in 2018 and finished up my dissertation while working over the next two years. My grad school research was unusually focused for a PhD student. All my research, starting with my first paper, was focused on answering the question of how to develop an AI that could beat top humans in no-limit poker. After we succeeded in that in 2017, my research shifted more toward generality and scalability.

My original plan was to defend in summer 2019, do an industry research stint for a year, and then start a faculty position in 2020. (1-year deferrals are common in academia these days.) So I applied to universities and industry labs in fall 2018. FAIR gave me an offer and also said that I could start working immediately, even though I told them that I'd be doing faculty interviews for most of spring 2019. That seemed like a strictly better option than staying in grad school and making near-minimum wage, so after considering a few other options I chose to join FAIR immediately.

I ended up liking it so much that I turned down my faculty offers and stayed at FAIR. Once I knew I wasn't going to faculty, there wasn't as much urgency to finishing my PhD. I wanted to include one more major project in my thesis, ReBeL, so I held off on defending until that was done. -NB

Actually the language model was capable of suggesting good moves to a human player *because* the planning side of CICERO had determined these to be good moves for that player and supplied those moves in an *intent* that it conditioned the language model to talk about. CICERO uses the same planning engine to find moves for itself and to find mutually beneficial moves to suggest to other players. Within the planning side, as described in our paper, we *do* use a finetuned language model to propose possible actions for both Cicero and the other players - this model is trained to predict actions directly, rather than dialogue. This gives a good starting point, but contains many bad moves as well, this is why we run a planning/search algorithm on top. -DW

A related question is "can CICERO take suggestions from other players?" to which the answer is "Yes!". CICERO uses its models to generate a list of "plausible moves" that it reasons over, but if someone suggests an unexpected move to CICERO, it will evaluate that move in its planning and play it if it's a good idea. -AL

AI sits at the very heart of work across Meta. We are part of Meta AI's Fundamental AI Research team -- known as FAIR. Exploratory research, open science, and cross-collaboration are foundational to FAIR efforts. Researchers like us have the freedom to pursue pure open science type work and collaborate with the industry and academia.

Research teams also work closely with product teams across Meta. This gives engineers an early view into where the latest in AI is heading and gives researchers an up-close look at how AI is working at scale. This internal collaboration has helped us build a faster research to production pipeline too. -AL

We disentangle the complexity of the action space from the complexity of the planning algorithm by using a policy proposal network. For each game state we sample a few actions from the network - sets of unit-order pairs - and then do planning only among these actions. Now, in case of continuous actions we will have modify the policy proposal network, but that was already explored for other games with continuous action space such as StarCraft. - AB

One challenge was being able to hold 6 simultaneous conversations at a human speed in the fast-moving "blitz" Diplomacy format, since CICERO has to do a lot of planning and NLP work for each message it sends (see Fig 1 in our paper). We ended up splitting CICERO into "sub-agents" that handle conversations with each other player. CICERO actually ran on 56 GPUs in parallel for our human games (although it can also run on a single GPU in slower time formats). -AL

I really strongly disagree that lying is a positive in Diplomacy. The best players do it as little as possible - it is a game about building trust in an environment where trust is hard to build. I think Diplomacy has a reputation for being about lying because new players think just because they can do it, they must. I am nearly certain that a "CICERO II" wouldn't lie more. -AG

[deleted]

I'm not entirely sure if this answers what you were asking, but on the strategic planning side of CICERO, in some sense the fundamental challenge of Diplomacy is that it has a large number of local optima, with no inherent notion of which optimum is better than any other. Because you need to sometimes cooperate with others to do well, the way you need to play depends heavily on the conventions and expectations of other players, and a strategy that is near-optimal in one population of players can be disastrous in another population of players. This is precisely what we observed in earlier work on No-press Diplomacy (Paper). Central to many of our strategic planning techniques in Cicero is the idea of regularization towards human-like behavioral policies, to ensure CICERO's play remains roughly compatible with human play, rather than falling into any of the countless other equilibria that don't. -DW

From a non-technical point of view, the fact that the human Diplomacy players we worked with (Karthik and Markus) were really excellent players so the model kept being evaluated against the best, rather than accounting for human players sometimes being average instead. Accounting for all levels of play was challenging! -AG

We tried hard in the paper to articulate the important research challenges and how we solved them. At a high level, the big questions were:

• RL/planning: What even constitutes a good strategy in games with both competition and cooperation? The theory that undergirds prior successes in games no longer applies
• NLP: How can we maintain dialogues that remain coherent and grounded over very long interactions
• Joint: How do we make the agent speak and act in a “unified” way? I.e. how does dialogue inform actions and planning inform dialogue so we can use dialogue intentionally to achieve goals?

One practical challenge we faced was how to measure progress during CICERO’s development. At first we tried comparing different agents by playing them against each other, but we found that good performance against other agents didn’t correlate well with how well it would play with humans, especially when language is involved! We ended up developing a whole spectrum of evaluation approaches, including A/B testing specific components of the dialogue, collaborating with three top Diplomacy players (Andrew Goff, Markus Zijlstra, and Karthik Konath) to play with CICERO and annotate its messages and moves in self-play games, and looking at the performance of CICERO against diverse populations of agents. -AL

Nah CICERO is still invited to the house games -NB

From a strategic perspective, it attempts similar things but the results are a little different - which is understandable as it reacts differently. It tends to build more unorthodox alliances just because it doesn't know they're unorthodox. It actually made the self-play games quite fun to watch, although if the point is to compete against humans it is kind of tangential to the key challenges. -AG

We tested the model using self-play frequently before we ever put it in front of humans (outside of our team). One interesting learning was that mistakes that the model makes in self-play games aren't reflective of the mistakes it makes when playing against humans. From a language perspective, in self-play, the model is more prone to "spirals" of degenerate text (as one bad message begets the next, and the model continues to mimic its past language). Moreover, humans reacted differently to mistakes the model made — in human play, a human might question/interrogate the agent after receiving a bad message, while another model is unlikely to do so. This really underscored the importance of playing against humans during development for research progress. -ED

I loved this problem! The average human player sends way too few messages compared to the best human players, so the challenge was how far to push this before it became.... weird. So it wasn't just infinite messaging either. I'll let others answer how that was technically achieved, but this was an underrated challenge to achieving great play. What a great question! -AG

Inhuman play would be flagged as untrustworthy and make it difficult for the AI to make alliances in game, thus leading to weaker play overall.

The title of the paper doesn't refer to CICERO being "human-like" necessarily (though it does behave in a fairly human-like way). Instead it refers to the agent achieving a score that's on the level of strong human players.

But also, CICERO is not just trying to be human-like: it’s also trying to model how *other* humans are likely to behave, which is necessary for cooperating with them. In one of our earlier papers we show that even in a dialogue-free version of Diplomacy, an AI that’s trained purely with RL without accounting for human behavior fares quite poorly when playing with humans (Paper). The wider applications we see for this work are all about building smart agents that can cooperate with humans (self-driving cars, AI assistants, …) and for all these systems it’s important to understand how people think and match their expectations (which often means responding in a human-like way ourselves, though not necessarily).

When language is involved, understanding human conventions is even more important. For example, saying “Want to support me into HOL from BEL? Then I’ll be able to help you into PIC in the fall” is likely more effective than the message “Support BEL-HOL” even if both express the same intent. -AL

In 2019 we had just finished up Pluribus and were discussing what to pursue next. We saw the incredible breakthroughs happening across the field, like GPT-2, AlphaStar, and OpenAI Five, and knew that we needed to be ambitious with our next goal because the field was advancing quickly. We were discussing what would be the hardest game to make an AI for and landed on Diplomacy due to its integration of natural language and strategy. We thought it could take 10 years to fully address, but we were okay with that because historically that kind of research timeframe had been the norm. Obviously things worked out better than we expected though.

Our long-term goal was always the full natural language game of Diplomacy but we tried to break the project down into smaller milestones that we could tackle along the way. That led to our papers on human-level no-press Diplomacy, no-press Diplomacy from scratch, better modeling of humans in no-press Diplomacy, and expert-level no-press Diplomacy. -NB

Re: Dialogue-related challenges: Moving from the "no press" setting (without negotiation) to the "full press" setting presented a host of challenges at the intersection of natural language processing and strategic reasoning. From a language perspective, playing Diplomacy requires engaging in lengthy and complex conversations with six different parties simultaneously. Messages the agent sends needed to be grounded in both the game state as well as the long, dialogue histories. In order to actually win the game, the agent must not only mimic human-like conversation, but it must also use language as an *intentional tool* to engage in negotiations and achieve goals. On the flip side, it also requires *understanding* these complex conversations in order to plan and take appropriate actions. Consider: if the agents actions did not reflect its conversations/agreements, players may not want to cooperate with it, and at the same time, it must take into account that other players might not be honest when coordinating/negotiating to plans.

Re: AI technologies in the future: advancements in this space have many potential applications and will hopefully improve human-AI communication in general to get closer to the way people communicate with each other. -ED

Re: memory of the whole game/chat — in terms of the dialogue, due to memory constraints, both our dialogue models and dialogue-conditional action models see a fixed context window (typically, only a few turns/phases worth of dialogue, depending on how many messages were sent in a given turn).

Re: betrayal/forgiveness — Many humans fall into the trap of trying to make another player lose out of ""revenge"", even at the cost of making bad strategic decisions relative to their own gameplay. CICERO is designed to take actions that are best for itself. - ED

The speed at which we managed to progress from no communication to full natural language Diplomacy also surprised the research team. When we started, the idea of an AI agent that could master no-press Diplomacy seemed like a multi-year effort, and the idea of an AI agent that could play full-scale Diplomacy in natural language seemed like science fiction. We thought it might take 10 years to reach this point. -NB

I think the speed that it went from playing no communication games to full communication games was the biggest surprise - not just the natural language but the adaptation of strategy and tactics. I expected it to really struggle to climb out of what it had learned from that style of game, but it did so pretty quickly, which is probably down to the technical expertise of the team. I guess beyond that, the AI plays some approaches that upset the inherited wisdom of the diplomacy playing group. I'm totally revisiting some opening lines for example. In terms of what we can learn - the strategic ideas the emerge seem to be very much aligned with high level human players. Patience, collaboration, improving position rather than brute force tactical tricks... at that level of abstraction it plays very similarly to a good human player. -AG

Thanks! Glad you enjoyed it! -NB

As we look at the incredible potential of what AI can unlock in the physical and virtual worlds we need to balance that optimism with an appreciation for the risks. These risks can come in many forms whether through unintended uses of new technologies or through bad actors looking to exploit areas of vulnerability. Being thoughtful about research release (through, e.g., special licenses, as you suggest), is one way to help this research move forward while limiting potential negative use cases. There are also many other research areas which I think are promising for bolstering positive use cases and limiting negative ones; to name just a few, improving control over language model outputs, investing in modeling for rapid adaptability and flexibility, discriminating between human and model-generated text, etc. -ED

[deleted]

CICERO always tries to maximize its own score. However, there is a regularizer that penalizes it for deviating from a human-like policy. When all actions have the same expected value (e.g., when it's guaranteed to lose no matter what) then it will just try to play in a human-like way, which may involve retaliating against those that attacked it. -NB

The next step is taking the lessons we've learned from CICERO and extending them more broadly to other research domains. We're also hoping that others are able to build on our open-sourced work and will continue to use Diplomacy as a benchmark for research. -NB

CICERO reasons about the beliefs, goals, and intentions of the other players. Whether that counts as "theory of mind" depends on the definition. This reasoning is partly implicit through the output of the policy network based on the conversations and sequence of actions, and part of it is explicit through the strategic reasoning algorithm. -NB

Not from the Meta team but you might want to take a look in the SM and search for "GPU"/"GPUs", they actually did a very nice job describing it (does not answer you question RE region but I thought it might be helpful, e.g. number of GPUs).

As someone without a PhD, I will say I definitely don't think it's necessary to have a graduate degree to work at the cutting edge of ML. Our team contains people with a mix of educational backgrounds working on all aspects of the projects, and the majority of the team do not have PhDs. I don't think there's an optimal choice for everyone, it probably depends on how you learn best and what type of problem you want to work on, but there's certainly a lot of great research being done by people without PhDs within industry! -AL

I don’t know about “biggest” :p but as someone without a graduate degree working in AI research, I’ve definitely felt imposter syndrome at times. One of the amazing things about working with large teams of research experts is that people bring extremely deep and diverse knowledge. Just on our team there are experts in NLP, reinforcement learning, game theory, systems engineering, and Diplomacy itself. When people are specialized in this way, the total knowledge on the team is much more than the knowledge of any individual, which is excellent for the team but was daunting for me at first! -JG

[AG] Me. The whole team is just next level and every day I was working with them I was sponging up ideas and knowledge. It's just so great being in a room with people who are so good at what they do. While I'm obviously OK at Diplomacy, the AI aspects and how the team attacked problems just blew my mind.

It takes significant effort, but yes, on the strategic planning side it is often possible to work out why CICERO came up with particular moves or intents. We often did this during development when debugging. You can look at the moves considered by the search for it and its opponents and see what values those achieved in the iterations within the search, and see how the equilibrium evolved in response to those values, you can look at the initial policy prior probabilities, and so on. Not entirely unlike walking through a debug log of how a chess engine explored a tree of possible moves and why it came up with the value it did. In fact, generally with systems that do explicit planning rather than simply running a giant opaque model end-to-end, it's usually possible to reverse-engineer "why" the system is doing something, although it may take a lot of time and effort per position. We haven't tried a human in the loop for choosing moves though. -DW

Generally crap player here who did better than Cicero twice and lost once.

I don't know what they would say but it looked to me like the matches in which they entered Cicero had a pretty wide range of skill, although I don't think there was anyone who diplomacy players consider to be at the top of the game. Hard to say as online diplomacy is so fragmented; Cicero was in a very new very niche variant.

Edited to add that I forgot about some of the players who were involved in the playtesting and design side. They're pretty good.

We joined a league designed by members of the active online Diplomacy community. The league included new players as well as more experienced players who have performed well in other Diplomacy tournaments. -AM

Figuring out how to get strong control over the language model by grounding in "intents"/plans was one of the major challenges of this work. Fig. 4 in the paper shows we achieved relatively strong control in this sense: prior to any filters, ~93% of messages generated by CICERO were consistent with intents and ~87% were consistent with the game state. As you note, however, the model is not perfect, and we relied on a suite of classifiers to help filter additional mistakes. Many of the mistakes CICERO made were relative to information that was *not* directly represented in its input (and thus required additional reasoning steps), e.g., reasoning further-into-the-future states or counterfactual past states, discussing plans for third parties, etc. We could have considered grounding CICERO in a richer representation of "intents" (e.g., including plans for third parties) or of the game state (e.g., explicitly representing past states), but in practice we found that (i) richer intents would be harder to annotate/select and often take the language model out of distribution and (ii) we had to balance the trade off between richer game state representation with the dialogue history representation. Exploring ways to get stronger control/improve the reasoning capabilities of language models is an interesting future direction. -ED

For me personally, I had no practical machine learning experience prior to 2017, although I did have experience in engineering, and with statistics and working with data. I often had personal programming projects going which I worked on in the weekends and evenings. But anyways, among these projects I picked an intro project that I thought would be fun (human move prediction with deep neural nets in computer Go), started looking up tutorials, academic papers, ML libraries and APIs, and that was the start of it. Pick something you're interested in, and dive in! -DW

JG: There's never been a better time to get into machine learning, with so many amazing open source projects being released, amazing blog and youtube tutorials, and communities of people trying to learn together. Whether you're interested in audio, image generation, game AI, or anything else, I'd recommend you clone a popular open source repo, play around with it for a while, and then see if you can make a small modification!

While many players do lie in the game, the best players do so very infrequently because it destroys the trust they’ve built with other players. Our agent generates plans for itself as well as for other players that could benefit them, and it tries to have discussions based on those plans. It doesn’t always follow through with what it previously discussed with a player because it may change its mind about what moves to make, but it does not intentionally lie in an effort to mislead opponents. We're excited about the opportunities for studying problems like this that Diplomacy as an environment could provide for researchers interested in exploring this question; in fact, some researchers have already studied human deception in Diplomacy: https://vene.ro/betrayal/niculae15betrayal.pdf and https://www.cs.cornell.edu/~cristian/Deception_in_conversations_files/deception-conversational-dataset.pdf. -AM

As noted in an answer to a previous question: we were originally targeting 24hr-turn games, but ended up pivoting to 5min-turn games due to the inability to gather a sufficient number of samples in the 24hr-turn format (as playing a single game can sometimes take months)! Playing 24hr-turn games would indeed pose additional challenges from a language generation perspective — while human players tend to send a similar number of messages in each format, messages in 24hr turns tend to be significantly longer (and likely more complex). Moreover, human players would have more time to interrogate mistakes from the bot, which could potentially lead to the agent making further mistakes. -ED

Regarding the post-game kibbitzing, we discussed this a few times, but every solution felt like we'd be faking it. For example, we could have put a human in the loop here but.... why? In the end we picked the most honest approach we could when dealing with the community, which was an ethical consideration that underpinned the whole project I think. -AG

One of the key challenges of Diplomacy is modeling how people might respond to your actions. We found that approaches used in prior game AI breakthroughs like Go and poker that relied purely on self-play were not able to anticipate "human" behaviors like retaliation. For that reason, a big contribution of our research is developing a way to incorporate human data into self-play, which allows us to find strong policies that also understand how people approach the game. -NB

Since this is a research effort, we don't have plans to host CICERO for public availability. However, we have open-sourced both the model files and code, which means you could host CICERO yourself on a private instance of webDiplomacy.net (also open sourced here). More details can be found here. -CF

Early on, we primarily evaluated the model using self-play, having team members play against it, and by building small test sets to evaluate specific behaviors. In the last year, we started evaluating the model by putting it in live games against humans (with another human in the loop to review its outgoing messages and intervene if necessary). We quickly learned that the mistakes the model makes in self-play weren't necessarily reflective of its behaviors in human play. Playing against humans became *super* important for developing our research agenda! -ED

I will be messaging you in 2 days on 2022-12-10 06:37:46 UTC to remind you of this link

2 OTHERS CLICKED THIS LINK to send a PM to also be reminded and to reduce spam.

^(Parent commenter can ) ^(delete this message to hide from others.)

There's quite a few open-source Reinforcement Learning challenges that you can explore with modest amounts of compute in order to build some experience training RL models, for example the Nethack Learning Environment, Atari, Minigrid, etc. For me personally, I had only worked in NLP / dialogue for years but got into RL by implementing Random Network Distillation models for NetHack. It's a fun area that definitely has its own unique challenges vs other domains. -AM

Our final agent does not explicitly try to detect deception. We do have models that predict the actions that people will play based on the board state and message history, and these models may implicitly detect betrayal by predicting actions that don't correspond with the message history. CICERO does have a model that tries to detect whether its *own* messages don't correspond to its intended action, and it will filter out the most egregious cases of that. -JG

ooh group chats would be fascinating

Actually our agent can play longer games, and much of our earlier testing (where we had to manually approve all outgoing messages) was on 24 hour games instead of the 5 minute games that we report on in the paper. The agent is overall a bit more effective in shorter time controls but the agent was in fact scoring quite well in longer time formats as well. However, these games take weeks to complete, and ultimately we decided that it would take too long to play enough games for statistical significance, hence the focus on shorter games. -JG

Great question! Back when we initiated Diplomacy, I hypothesized that "an agent that can read the rules of any game and play it at an intermediate level" would be the next challenge problem. There's been so much progress on the language modeling side that I think a system like this is within reach within the next 2-3 years if substantial effort was devoted to it. We're starting to see similar task-generality in large language models on real-world tasks, although constructing a symbolic representation for planning out of a text description is still an open research question! -AL

I think you could take a similar approach to Mafia or Among Us and do well. In fact, Mafia would be easier because it's still a two-team zero-sum game. We chose Diplomacy specifically because we thought it would be the hardest game to make an AI for and the most "real-world" game due to its natural language component. Now that we've achieved human-level performance in it, we're hoping to move beyond recreational games toward more real-world domains. -NB

The learned features are specific to the game of the Diplomacy because the data we used is specific to the game of Diplomacy, but the ideas can be transferred to other domains. Rather than just learning Diplomacy by playing against itself, the AI used a model trained on human games both to guide exploration during training (sampling moves from this model during self-play) as well as during planning (consider what actions humans are likely to take). It's not always obvious exactly how to apply this, but we think there's exciting opportunities for research in this space! -AM

We were originally targeting 24hr-turn games, but ended up pivoting to 5min-turn games due to the inability to gather a sufficient number of samples in the 24hr-turn format (as playing a single game can sometimes take months)! Playing 24hr-turn games would indeed pose additional challenges from a language generation perspective — while human players tend to send a similar number of messages in each format, messages in 24hr turns tend to be signficantly longer (and likely more complex). Moreover, human players would have more time to interrogate mistakes from the bot, which could potentially lead to the agent making further mistakes. -ED

Controlling the dialogue model via intents/plans was critical to this research. Interfacing with the strategic reasoning engine in this way relieved the language model of most of the responsibility of learning strategy and even which moves are legal. As shown in Fig. 4 in the paper, using an LM without this conditioning results in messages that are (1) inconsistent with the agent's plans, (2) inconsistent with the game state, and (3) lower quality overall. We did not conduct human experiments with an LM like this or a dialogue-free agent, as such behavior is likely to be frustrating to people (who would be unlikely then to cooperate with the agent) and quickly detected as an AI. -ED

[deleted]