Comments

You must log in or register to comment.

gargolito t1_jc889m6 wrote

so... what are perplexity filters ?

2

usc-ur OP t1_jc9ss05 wrote

A perplexity filter allows to remove sentences by likelihood to a given language model. In there you need to "play" with the parameter or threshold

3

Aphix t1_jca204m wrote

Mind elaborating?

2

usc-ur OP t1_jca4ot7 wrote

Sure! The idea is that you create a language model from a given corpus (let's say BNC) and then you use a similarity measure, in this case, perplexity, but can be another one to test how well your sample (sentence) "fits" into the model distribution. Since we assume the distribution is correct, this allows us to identified malformed sentences. You can also check the paper here: https://www.cambridge.org/core/journals/natural-language-engineering/article/an-unsupervised-perplexitybased-method-for-boilerplate-removal/5E589D838F1D1E0736B4F52001150339#article

2

Aphix t1_jc739fl wrote

Link to GitHub?

1

usc-ur OP t1_jc73mgl wrote

https://github.com/citiususc/pyplexity . My bad :)

2

Aphix t1_jc799qb wrote

All good, might want to link the other ones you posted in those other post comments, too (Smarty-GPT, etc) -- most of the time you can just link the GitHub directly from the post here.

2

usc-ur OP t1_jc7guwj wrote

>All good, might want to link the other ones you posted in those other post comments, too (Smarty-GPT, etc) -- most of the time you can just link the GitHub directly from the post here.

My bad, quite a newbie: https://github.com/citiususc/Smarty-GPT

2