You mean you don't put zucchini on your pizza???

You can only communicate with a person in a box through note cards written in Mandarin (or whatever language you prefer). You put one in, you get out a response. It would be safe to assume the person in the box understands Mandarin. In reality, they have a reference book (of arbitrary size) that simply has the response message for any message it receives. They have absolutely no idea how to understand the messages (if they're true, make sense, or really any context about them at all).

It's not "true" intelligence. We have fact systems all over the place, and this isn't that - there's a reason these AI are terrible at math, it's deterministic - and ChatGPT et al is not yet smart enough to parse that kind of context from what you're giving it, even to the level of "what is 2+2".