Code quality of my own and my team's code

The reliability of the engineering platforms we use. (spark, gpu clusters, ci/cd build pipelines)

The correctness and completeness of the data we ingest

RWKV is recurrent right? Why is it token limited?

Because the algorithm seems to only work on inference. Probably due to memory management of the cached activations or something. (Idk the actual technical reasons)

Fast inference is also important to anyone who wants to deploy these things.

Basically I would just make sure the metrics being compared are computed the same way. Same numerator and denominator like summing vs averaging, over the batch vs epoch. If the datasets are the same and the type of metric you are computing is the same it's comparable.

The implementation details just become part of the comparison.

Not exactly. In batch RL the data they train on are real (state, action, next state, reward) tuples from real agents interacting with real environments.

They improve the policy offline. In RLHF there actually is no env. And the policy is just standard LLM decoding.

I find it a little weird that RLHF is considered to be reinforcement learning.

The human feedback is collected offline and forms a static dataset. They use the objective from PPO but it's really more of a form of supervised learning. There isn't an agent interacting with an env, the "env" is just sampling text from a static dataset and the reward is the score from a neural net trained on a static dataset.

People are rushing to deploy LLMs in search, summarization, virtual assistants, question answering and countless other applications where correct answers are expected.

The reason they want to get to the latent space close to the answer is because they want the LLM to output the correct answer.

The source I linked in the comment you linked and then deleted.

I think it's not suitable because it isn't really related to the process of a machine learning anything. It seems to me to belong to the field of human computer interaction.

That comment is very over the top sarcasm. You would have realized that if you had checked the source I linked.

may be an unpopular opinion these days but I don't think prompt engineering is a suitable topic for /r/MachineLearning

I just have one dataset too. I train, pick hyperparameters, and test on the same data. Nobody can get better metrics than me. :D

The methods and models are identical yep. It's basically just to denote that whether the labels were assigned by a human or determined automatically.

along with each prompt, just put: "And at the end of your response, state on a scale from one to ten how confident you are in you answer"

This works amazingly and is very accurate. source

It has the added bonus where you can get confidence intervals on your confidence intervals just by asking how confident it is in it's estimation of its confidence.

Have you tried it with say an MLP or small convnet on cifar10? I think that would be the next logical step.

> test loss decreased

What function are you using to evaluate test loss? cross entropy or this norm function?

That's not reasoning. It's spitting out semi-random moves. If you keep giving it more and more chances it increases the probability of getting a set which has some legal moves.

Microsoft paid 1B to use GPT3.

That's an interesting topic that I think deserves further investigation. On the surface it sounds like the size of the LM impacts the mechanism by which the LM is able to "secretly perform gradient descent".

Is finetuning similarly unstable for small sized LMs?

If the goal is the mechanism rather than the performance why tune the seed for performance in the first place? The examples used doesn't change the mechanism.

I have an issue with the experiments.

> For ICL, we fix the number of demonstration examples to 32 and tune the random seed for each task to find a set of demonstration examples that achieves the best validation performance. For finetuning, we use the same demonstration examples for ICL as the training examples and use SGD as the optimizer

They go through a set of random seeds to pick the "best" possible samples for in context learning, and then use the same set of examples for fine tuning. I think this biases the results in favor of in context learning.

A more fair way to do this would be to use a truly random set of examples, or to use use the same approach and tune the seed to find the "best" set of examples for finetuning as well.

for datasets with ~1000 rows and 10 columns iirc

Quick poll to see what the opinions are here:

Is it ok to tune the random seed and pick the seed which is best on the validation set?

Thanks for the reply and the resource! You're right about the relatively recent influx of people who enter the ML field via deep learning first. Seems like most of the intro material focuses on logistic sigmoid based methods.

That said, do you think there is a fundamental reason why other log likelihood based methods such as probit and poisson as you mentioned haven't caught on in the deep learning field? Is it just that probit doesn't give an edge in classification, and such a large portion of use cases don't require anything besides a classification based loss?