Viewing a single comment thread. View all comments

planetofthemapes15 t1_jeg1iqc wrote

Cool, I had a mental model very similar to this which I was planning on implementing next week. I'll just try yours and if I make an improvement I'll submit a PR.

4

KD_A OP t1_jeghvnn wrote

Yeah I was surprised that this wasn't already coded up--it's been 3 years since we've found out that sampling from GPT-3 is a good zero-shot text classifier.

While benchmarking this method on the infamous Winograd Schema Challenge, I ended up finding a 2018 paper^1 w/ pretty much the same idea as CAPPr. The only difference is that CAPPr typically transposes that probability, and it naively incorporates a prior.

  1. Trinh, Trieu H., and Quoc V. Le. “A simple method for commonsense reasoning.” arXiv preprint arXiv:1806.02847 (2018).
3

nbviewerbot t1_jeghww5 wrote

I see you've posted a GitHub link to a Jupyter Notebook! GitHub doesn't render large Jupyter Notebooks, so just in case, here is an nbviewer link to the notebook:

https://nbviewer.jupyter.org/url/github.com/kddubey/cappr/blob/main/demos/wsc.ipynb

Want to run the code yourself? Here is a binder link to start your own Jupyter server and try it out!

https://mybinder.org/v2/gh/kddubey/cappr/main?filepath=demos%2Fwsc.ipynb

^(I am a bot.) ^(Feedback) ^(|) ^(GitHub) ^(|) ^(Author)

2