Comments

You must log in or register to comment.

__Maximum__ t1_j2n9cnv wrote

Everyone is being sarcastic here. In reality, we all pray while checking the loss every 20 seconds.

130

vandelay_inds t1_j2p35i5 wrote

Really, though, I think being really disciplined about this habit is important, because it’s so easy to get sucked into. It’s like a little shot of dopamine every time that little number on the screen ticks upward. Makes me feel like I’m on Wall Street. Haha

17

matigekunst t1_j2mitzy wrote

I train whenever the machine trains. Whenever I put on a run I do a set of squats or pushups. I see it as a regularisation method. Forces me to think a bit more before turning on a big run

74

hollow_sets OP t1_j2muixs wrote

Doing this for a model that has an ETA of 5 days sounds a bit too much for a workout ;-;

16

matigekunst t1_j2mv9ak wrote

I mean just do a set whenever you turn on the run. If you're anything like me you turn on tons of runs where you quickly spot a mistake.

Besides that, I would also recommend exercise every day. Not 120 hours long of course, just 30 minutes to an hour. I used to have very bad slouching posture like a lot of my colleagues. ML researchers spend a lot of time sitting behind a desk. Bouldering and going to the gym have wonders to my posture

10

hollow_sets OP t1_j2mvkr0 wrote

Hahaha yeah I do spot mistakes as soon as I start the run

That sounds like a good idea A set or maybe just read a paper while it trains

Edit: Also I'll be going back to my University campus(still a bachelor's student) this week so physical activity is going to go off the charts

4

eldenrim t1_j4ks6ei wrote

I am starting bouldering soon. What's your routine on a given week given gym as well?

1

adt t1_j2mcj90 wrote

5 days is pretty good! Some of the big models are many, many months.

Maybe you'd enjoy reading Meta's OPT-175B logbook while you're waiting...

https://github.com/facebookresearch/metaseq/blob/main/projects/OPT/chronicles/OPT175B_Logbook.pdf

22

hollow_sets OP t1_j2mh63j wrote

Oh wow thats actually pretty good
I'll read it and also start maintaining my own log book too

Since I am working on two research projects, This will be fun

8

vidret t1_j2meexa wrote

Play Elden Ring

And quadruple check everything

17

hollow_sets OP t1_j2mh9yh wrote

Well well guess what I just got a signal kill at evaluation hohoho (I am evaluating the model after every 1000 steps and its takes an approximate 5 hours to go through each 1000 steps) This was the first eval check so fuck

4

Competitive-Rub-1958 t1_j2mxgnz wrote

use `%` modulo to do a eval check before you start training (i.e 0th step). Saves a ton of time to debug, because something always goes wrong.

9

hollow_sets OP t1_j2mzo2p wrote

Yea, thanks for the advice :D (I was going to wait like an idiot) Fixed it now and seems like it is running properly

3

JustOneAvailableName t1_j2nhjks wrote

Personally I also like to eval way more often than every 5 hours. Perhaps use a smaller eval subset for every hour?

4

hollow_sets OP t1_j2nirn7 wrote

Sounds fair enough Current evaluation time is like 1.5 hours so I didn't go ahead with an hourly evaluation plan

3

sensei_von_bonzai t1_j2n2frn wrote

100 crunches everytime your validation error increases. You’ll either have perfect abs or great models

16

hollow_sets OP t1_j2n3a8o wrote

Holy moly, Guess what, I am doing this now
Going to participate in kaggle competitions more often (I have a fear of competitions so I never participated) and everytime I fuck up
I increase the number of crunches by 10

3

Tgs91 t1_j2n5aih wrote

Are you in academia or industry? In industry, I do other work while I wait for training to complete. Code cleanup, refactor and simplify my modules so they'll be easier to maintain, start building out modules for post processing / integrating the model for the end use case. If all of that is already completed, I start working on another project in my teams backlog. There's always other work to do, no reason to sit around waiting for a model to train.

15

hollow_sets OP t1_j2n6nmg wrote

Academia for now
Since Im a student (bachelors) and no one wants someone with just a bachelor's so I can't really enter the industry properly even if I want to

1

Tgs91 t1_j2na3mm wrote

As a student, you should take the time to work on code cleanup. Usually I see students use one big training script that has a lot going on. For my projects I typically build out a pip installable module with submodules for preprocessing/structuring raw data, model building with lots of kwargs so it can be customized, dataset objects with transformations or randomness etc for batch loading efficiently, etc etc. My actual training scripts are only a few lines of code. Hyperparams in all caps at the top, import functions from my module, and call the functions. And my modules are written in a way that employees of various skill levels can contribute to the project. Myself and another colleague do all of the more advanced AI work, but any member of the team can be a USER of the module, and we have more general data scientists that can contribute to preprocessing code, containerization, post processing tools, etc.

Even if you don't do a full module, make a utils.py file to pull out any long pieces of code and write it as an importable function. Use docstrings for every function with Google's docstring style guide (or use the autodocstring extension on VSCode, it's great). Use a linter like flake8 or black to make sure your code looks clean and professional. This all seems like minor, tedious stuff, but if you have to go back and edit/maintain code you wrote a year ago, it's a lifesaver. And it also means that in an industry environment, another coworker can step in and easily understand and edit your code. It might not make a functional difference to you right now, but good, clean, professional code is great on a resume.

9

hollow_sets OP t1_j2ndrb9 wrote

This sounds like a good plan to do while I wait for the model to train.

I'll start from tomorrow (since its 10:30 pm and I feel like I have burnt myself out for the day fixing the errors) Hope no more errors pop up while I sleep

2

PredictorX1 t1_j2n5fy5 wrote

I do the dishes, change batteries in things that need them and waste time on Reddit.

7

bitemenow999 t1_j2n4e7k wrote

I start writing the paper...

4

hollow_sets OP t1_j2n6i6u wrote

I would have as well but currently I have no clue what to do
Right now I am training the model(efficient-video-recognition) just to see if its resource friendly to our servers or not

So no clarity in which direction I have to move.

1

bitemenow999 t1_j2n8vw7 wrote

you can always use a smaller dataset and scale down the model to ensure it works and then train it whole, at least this is what I do... Generally waiting a week to see if the model works is very time-consuming...

3

tripple13 t1_j2ohf7r wrote

I continue working on that long backlog of things I'd like to implement:

  • Additional models (change of encoder/decoder for future runs)
  • Additional loss parameterisations (because you can never get enough)
  • Additional dataloaders for the inclusion of more datasets (because without killing penguins, no paper)
  • Additional bug-squashing/re-factoring which I've put off using TODOs as comments odd places in my code
4

gradientpenalty t1_j2pfy52 wrote

I try to get to sleep, but I can't
cause each time I woke up, the loss went to nan

I try to hang out with friends, but
every 5 minutes I kept refreshing weights and biases and my friends never hang out with me anymore

I try to play online games, but
each time the training went into OOM, I just close the game and try to tune my hyperparameters.

Now I just pray anxiously while scrolling online store for second hand 3090. And I look more or less like Gollum

4

hollow_sets OP t1_j2pggqr wrote

Gah never had friends in the first place to allow them to leave (joking)

But yeah gotta tune it I guess loss went to nan and I just woke up ;-;

2

ThickDoctor007 t1_j2nkqb2 wrote

I go running if I manage not to look at error charts in tensorboard

2

hollow_sets OP t1_j2nl7d2 wrote

Hahahaha I use weights and biases for this It just tells me the error rates and accuracy in a graph and if the program is running or not

Other than that I just stay in the illusion that everything is fine

1

thePsychonautDad t1_j2ntiov wrote

Pina colada & a book, it goes great with the tropical temperature in the office during training. It makes a Canadian January feel like the Bahamas in August.

2

Hyper1on t1_j2om5c4 wrote

This is why having multiple projects is good. Just work on other coding or writing up while you wait.

2

Rnk_007 t1_j2qnoz4 wrote

Touch grass.

2

RedditRabbitRobot t1_j2qsiq2 wrote

Can you parallelize runs ?

2

hollow_sets OP t1_j2yav9l wrote

Parallelize as in? Multiple models running together?

1

RedditRabbitRobot t1_j2z7bqa wrote

multiple trainings of different models or same model with different training parameters/hyperparameters.

i.e if you have a cloud environment with a number of processing units available on demand.

1

[deleted] t1_j2n1w01 wrote

You mean computing wise? Well I guess work on my procedural generation engine, listening to music, hunt for software vulns with SMT solvers and such. Or go about an IRL activity/ hobby.

1

hollow_sets OP t1_j2n3eff wrote

Ohh that sounds fun
I'll try working on some full stack development as well now that I have free time

2

[deleted] t1_j2oosu9 wrote

Yeah, mostly is fun - more so even lately chatting with ChatGPT as coding assistant simultaneously. What will you be working on more specifically :) ?

2

hollow_sets OP t1_j2pgc0u wrote

Im working on video recognition Just started the project though so no direction as of now

2

Clicketrie t1_j2tf1q4 wrote

I always retrained my models on Fridays so that I'd lose less time. Then I found that I always had enough administrative type tasks, meetings I'd put off, or training I'd like to do (training that I totally wouldn't take advantage of if it wasn't for the downtime), to get me to the end of the model training.

Just another thought, have you considered creating content for Reddit or LinkedIn? Obviously you created this post, but I mean educating others about what you're working on? My network is a huge piece of my career now, if I'm looking for a job I'm not applying places because I get a lot of inbound opportunities. It's tangential to work and will help you in your career (As long as you're not in finance or some other industry where people don't talk about their work).

1

hollow_sets OP t1_j2w9czc wrote

I do post alot of random stuff on twitter and sometimes LinkedIn as well

I am considering to make content on what I am working on (atleast on streams I do show what Im doing to the few viewers I get)

Might consider posting on reddit too but not LinkedIn for explaining what I'm doing (since there are some real assholes in my university and I don't think I want them to know what I'm doing in work)

(Also would it be possible for to send some opportunities to apply to? My career path sure is going to be in the ML development industry and later on to academia)

1

Clicketrie t1_j2wigll wrote

Sure. If I see another MLE role I can share.. they always ask if I know anyone.

2

hollow_sets OP t1_j2wkvlb wrote

Aye thats going to help me out alot

Also which subreddits would you recommend me for posting about what I am doing? I guess writing blogs on medium is one way (Also I started to maintain a log book for my on-going project, though still local) On LinkedIn, its more like just normal text posts which I don't think is possible to do on reddit.

1