[D] Mtruk alternatives for extracting information out of text

Submitted by elcric_krej t3_10augmv in MachineLearning

I need some validation samples for an information extraction task, basically extracting a list of objects with 4 fields from a text (+ a binary flag).

I intended to use mturk for this, but they seem to have some billing issues and I haven't managed to have them allow us to actually spend any money in a week.

I've looked at a few alternatives but most seem very small and focused on simple tasks and surveys. Have any of you successfully used something other than mturk for this kind of task?

2

Comments

You must log in or register to comment.

zaptrem t1_j47af3e wrote

If it’s just text have you tried using an LLM like GPT3 with human verification?

−2

elcric_krej OP t1_j49jb2z wrote

Not only have I tried, this is precisely what I am doing, human verification is the exact use case for a service like mturk :)

1

zaptrem t1_j49jqfr wrote

Idk why I’m being downvoted… you said you “wanted validation samples” not that you wanted to validate samples.

1

elcric_krej OP t1_j4a7zkl wrote

Not the one doing the downvoting, but, isn't that same thing?

Validating a sample and having validation samples, when your problem has a "known solution" or is close to, is equivalent.

1

zaptrem t1_j4a83db wrote

Not your fault, but it seems reasonable to interpret "validation samples" as samples for your validation set (e.g., you wanted people to manually label something for your training/validation data).

1

elcric_krej OP t1_j4atwg1 wrote

Yes... that's the correct interpretation, hence why I need mturk, to get people to manually label (well extract) something from my training/validation data.

​

I'm still rather confused about where the misunderstanding is.

1

zaptrem t1_j4c9p2l wrote

Validate samples = you feed completed outputs to humans and ask them to click correct/incorrect.

Validation samples = you all humans to manually complete the task themselves. Slow and expensive.

1