Viewing a single comment thread. View all comments

zaptrem t1_j47af3e wrote

If it’s just text have you tried using an LLM like GPT3 with human verification?

−2

elcric_krej OP t1_j49jb2z wrote

Not only have I tried, this is precisely what I am doing, human verification is the exact use case for a service like mturk :)

1

zaptrem t1_j49jqfr wrote

Idk why I’m being downvoted… you said you “wanted validation samples” not that you wanted to validate samples.

1

elcric_krej OP t1_j4a7zkl wrote

Not the one doing the downvoting, but, isn't that same thing?

Validating a sample and having validation samples, when your problem has a "known solution" or is close to, is equivalent.

1

zaptrem t1_j4a83db wrote

Not your fault, but it seems reasonable to interpret "validation samples" as samples for your validation set (e.g., you wanted people to manually label something for your training/validation data).

1

elcric_krej OP t1_j4atwg1 wrote

Yes... that's the correct interpretation, hence why I need mturk, to get people to manually label (well extract) something from my training/validation data.

​

I'm still rather confused about where the misunderstanding is.

1

zaptrem t1_j4c9p2l wrote

Validate samples = you feed completed outputs to humans and ask them to click correct/incorrect.

Validation samples = you all humans to manually complete the task themselves. Slow and expensive.

1