Viewing a single comment thread. View all comments

ARGleave t1_iuq67g8 wrote

Reply to comment by ThatSpysASpy in [N] Adversarial Policies Beat Professional-Level Go AIs by xutw21

I replied to this in https://www.reddit.com/r/MachineLearning/comments/yjryrd/comment/iuq5hq9/?utm_source=reddit&utm_medium=web2x&context=3 but since this is currently the top-voted comment, I wanted to be clear that the scoring used is Tromp-Taylor, which KataGo was primarily trained with and which is the standard for evaluation in Computer Go.

Good point about the regularizer! KataGo does indeed have some functionality to encourage what it calls "friendly" passing to make it nicer for humans to play against, as well as some bonuses in favour of passing when the score is close. We disabled this and other such features in our evaluation. This does make the victim harder to exploit, but it's still possible.

I think it's reasonable to view this attack as a little contrived, but from a research perspective the interesting question is why it exists in the first place -- why didn't self-play discover this vulnerability and fix it during training? If self-play cannot be trusted to find it, then could there be more subtle issues.

26

picardythird t1_iuqhsya wrote

It is absolutely misleading to claim that Tromp-Taylor is "the standard for evaluation" in computer go.

Tromp-Taylor scoring has been used occasionally as a convenient means of simplifying the way that games are scored for the purposes of quantitative evaluation. However, area scoring (such as standard Chinese rules) or territory scoring (such as standard Japanese rules) are overwhelmingly more common, not to mention that these are actual rulesets used by actual go players.

Your claims are inflated and rely on overly-specific problem statements that do not map to normal (or even common) usage.

−7