[D] Giving Up on Staying Up to Date and Splitting the Field

Submitted by beezlebub33 t3_xxyeun in MachineLearning

Does anyone else feel completely unable to keep up with machine learning and AI in general? I have my sub-sub-field and I do my work in (applied, mostly) and I read those papers, but I at least try to keep somewhat up to date on the entire topic of machine learning.

I mean, at this point I understand Transformers and related, and I kind of understand Latent Diffusion Models and Graph Neural Networks but not enough to use them, but I've lost the bubble on what's happening in deep reinforcement learning. I'm sure AlphaTensor is great, but I just don't have the time and energy.

I'm dreading NeurIPS and trying to figure out what people are talking about. I am wondering if ML needs to do what physics did a while ago, and just give up on trying to understand all of it.

I have a relative who does physics of solar cells (something about hot carriers and hyperfine states???) who doesn't understand what the relativity people he went to undergraduate with are talking about. They go to different conferences now.

268

Comments

You must log in or register to comment.

3jckd t1_ireitny wrote

Why do you even need to understand everything? FOMO?

Either you work on fairly general topics, or more generally applicable things, and then you don’t need to read a bazillion papers that propose yet another flavour of attention.

Or you work on something more specific within one domain, e.g. NLP, and then you don’t need to know the details of e.g. image generation that you brought up.

Pro-tip: high level understanding of the field as a whole, and solid understanding of your specific niche is what you’re after.

231

GFrings t1_irelmbh wrote

This. We need to start treating ML as a collection of fields with synergies. Just like they teach in engineering school, you're going for a T shaped skill tree. Be generally knowledgable about what's out there, but do a deep dive of one particular niche that interests you.

Also, it should be said that if you're not a researcher, one of the hardest parts of AI is the data. Everyone gets this wrong, particularly when they start chasing the latest backbone, or loss function, or a hundred things that see dozens of publications per week on. The ML engineering around standing up a good experiment, regardless of what you're actually putting into the model slot of this experiment, is where 90% of the effort should be going.

82

beezlebub33 OP t1_ireprux wrote

>Why do you even need to understand everything? FOMO?

FOMO is part of it. I'm interested in ML in general, and think that AI is coming (really unsure when) and I want to be part of all of it. I have this fear that someone is going to do something very important and I just won't know about it.

There is also the expectation (at least where work) that when you are a 'Machine Learning Engineer' that you have a pretty good grasp on the field as a whole. You don't want someone to say 'What do you think of Random Forests, how do they work?' and you go 'What's a Random Forest?' (hyperbolic example). I kind of sucks when you are the 'ML Guy' in a pretty big company so people come to you with (random) questions.

Finally, I'm old enough that I think that when ML started, I did know most of it. Having been raised on Duda and Hart and then was around when backprop made it's second renaissance, I remember when Elements of Statistical Learning came out. So, perhaps it's a sadness that the field has blossomed beyond me.

34

nullbyte420 t1_irethse wrote

To answer questions from normal people you don't need to know the latest thing. You need to understand the basics really well, because that's what they don't 🙂 Anyone can read the latest paper but not everyone can put it into context and compare it to existing models.

22

csreid t1_irg0y7f wrote

I kinda get where OP is coming from, though. With all the pop-sci ML stuff and big press releases for popular consumption hitting really shortly after actual publication, there's always a risk that some manager will be like "hey I just read about stable diffusion on Twitter, can we use it to do this?" and then you're a deer in headlights bc you weren't at the press conference where they introduced it and you have no idea what the manager is even talking about.

2

Ulfgardleo t1_irg3wty wrote

"This is a fairly new model and I do not know the details. If you seriously consider this, I can read up on the most recent work and then we have a meeting next week and discuss whether and how it could help us".

The awesome thing about solid basics is that you can do exactly this.

20

csreid t1_irjtm3b wrote

But this bit:

>"This is a fairly new model and I do not know the details"

is hard! I understand having anxiety about being The ML Guy and not being able to immediately answer questions.

2

Ulfgardleo t1_irm2bsq wrote

yes. I think at this point it is important to realize that in the exact moment you got hired by a company, your role changed.

You were the guy with a PhD straight from university who did top-notch research. Now, you are the guy hired to make this project work.

If your job description does not include "active research" or "follow the most recent advances in ML research" then it is not your job to know what is up - especially if it is an advancement in a subfield of ML your project is not actively interested in.

2

AliceHwaet t1_irh3fnq wrote

It's not necessarily FOMO. look at job postings these days. They want the kitchen sink, the counter it was built into, the floor plan, the carpenter who did the rough framing of the whole damn house, how did you acquire the property to build the house, where did you get the money, prove it, and on.

Someone newish to the field thinks all these things are important because job posters just throw every damn AI/ML skill they've ever heard of or googled into the JD.

That needs to stop and the person describing the T method is right. Business and hiring managers have it bass ackwards.

7

new_name_who_dis_ t1_irepq7z wrote

> I am wondering if ML needs to do what physics did a while ago, and just give up on trying to understand all of it.

I think a lot of people in ML already have been doing that. This doesn’t need to be a widely acknowledged shift. You research what you’re interested in, that’s how you specialize.

59

there_are_no_owls t1_irf92we wrote

I agree with what you said. Still there remains the question of whether to split conferences. What's your take on this?

4

new_name_who_dis_ t1_irf9n5b wrote

Aren’t there already a lot of specialized conferences? Like cvpr and iccv are computer vision, I know theres an NLP one I don’t remember the name.

RL is the only one that I can’t think of a famous specialized conference for but there probably are ones that just aren’t as famous.

12

MrAcurite t1_irfag9x wrote

"Computer Vision" on its own is already a fucking massive field though, covering generative modeling, scene understanding, edge hardware, and everything in between.

8

csreid t1_irfzua0 wrote

Generative image models probably need to fork off sometime soon, especially text-guided versions. It's a pet peeve of mine that we're calling it "vision". Vision, at least to me, implies seeing/making sense of what is actually there.

1

Ulfgardleo t1_irg49u1 wrote

While text-guided image generation is flavour of the month, i don't think that it has broad enough impact to generate a consistent large enough amount of papers to be able to sustain its own conference.

4

eliminating_coasts t1_iri5g4z wrote

I suspect automated image generation is going to become an industry, if not a separate field of study, within the next three years, so we'll start seeing hybrid programmer/artist conferences springing up, with people leaning more towards demonstrations rather than papers.

1

Best_Mord_Brazil t1_irfl3if wrote

NLP has several large conferences that people publish impressive research at.

ACL, EMNLP, COLING, LREC (for datasets!), and the various geolocated variants of each are all prestigious venues.

2

zy415 t1_irh6dg3 wrote

Nowadays there are many niche conferences in different subareas of ML. However, I think it will take some time for those niche conferences to be "mainstream" in the specific subarea, because prestige matters (especially in job application) and those niche conferences are just not as prestigious as the top ML conferences, at least for now. See here for a discussion: https://www.reddit.com/r/MachineLearning/comments/vjqdom/d_niche_ml_venues_vs_top_ml_conferences/?utm_source=share&utm_medium=web2x&context=3

2

taplik_to_rehvani t1_ireobti wrote

I have burned out trying to chase all the new papers. Recently I have just given up. I guess age is catching up but I want to take a break and just focus rather than reading papers all day long.

27

jnwatson t1_iresi9d wrote

Lol I remember early in my career trying to keep up with all of CS before I gave up sometime in the late 90s.

CS in general is moving so fast that it is hard to keep up with what all the sub fields even mean now.

ML has hit a critical juncture and is moving at light speed. I’d be surprised if any mere mortal can keep up with all of ML now.

23

beezlebub33 OP t1_ireyurk wrote

I understand that. I don't even bother understanding what the latest developments in javascript libraries are or what release we're in with Java. I can barely keep up with changes in Python, ML libraries, and things specific to my work. Since (as I mentioned) I do applied stuff, I have to go between handling large data (dask, BigQuery, etc.) and ML frameworks (sagemaker, vertex ai, etc.) as well as the underlying algorithms. It's too much.

5

PLxFTW t1_irfjabg wrote

I literally don’t care anymore. In school I felt like I needed to keep up on everything but after working for 3 years now I just don’t try. I do research related to my work, and live my life outside of work. I have literally zero interest in ML beyond work and even during work I have more pressing concerns.

17

tiwanakota t1_irfy66a wrote

That's sounds... Boring

−4

PLxFTW t1_irfynm6 wrote

That’s how I feel about 99% of the research field. It’s all the same bullshit over and over.

15

bobwmcgrath t1_irerwsl wrote

I feel the opposite. While there is a lot more cool stuff out there then there was 10 years ago when I started. The work has not changed much. Everything that's interesting gets provided for free from universities, or google. This leaves me with the chore of sorting through large piles of data. Honestly the ai portion of every project I have been on is tiny compared to the overall infrastructure required while management only cares about what's new and shiny.

16

nelsonkrdenas t1_irer2zd wrote

If we assume AI is a tool to solve problems and it's more prevalent as the years go by, we are not going to keep up with the whole new research, because specialists in many different industries will create models for specific purposes. We can try to be generalists, but just one human doesn't have enough time to consume the content generated by thousands. In the end, we can try to learn general models, have a superficial understanding of promising new techniques, and learn about a handful of specific problems to be a specialist in those.

10

Rebombastro t1_iricpkn wrote

Exactly. Being an expert in every single field of your profession is never a requirement. The need to be one is more of a sign of fomo or maybe even some underlying psychological issue.

3

CashyJohn t1_irfq3lq wrote

It’s honestly not that difficult to keep up with the significant improvements in deep learning from a fundamental research perspective. AlphaTensor, amongst many others, is a cool application of reinforcement learning. It’s implications are huge by any standards but it’s not the discovery of a new model. Although diffusion models are also not new outside of ml, they recently started to gain attention as a new model type to train. For me, this is what’s interesting: new types of models, new ways to use SGD in a DL setup. CNNs, RNNs, Att,… are fundamental computation models. Extensions, improvements and upscaling is what I see in 99.99% of papers. GAN is a new model, VAE is a new model, actor critic is a new mode, etc. when it comes to fundamental dl research there are not many of the big ones imho.

7

wintermute93 t1_irfnmrx wrote

Who in the world is expecting you to stay on top of every part of the field? That's not how any technical discipline works. It's not even how CS works.

6

Extreme_Photo t1_irfj0wf wrote

I made a list of my "obsessions" which is where I will focus. I try to stay inside that list though I reevaluate it quarterly. I also use a knowledge management system (Obsidian) where I store summaries of what I read so I know I can find something again if I need it. These two habits have helped me.

5

LevKusanagi t1_irfhkne wrote

if you work in applied ml, let the project scope or expected near future project scopes dictate the locus of your learning. learn the unchanging principles (how to diagnose errors, and what to do about that) and then ride the wave instead of trying to drink it. if this is too vague please ask me more
Edit: i think the scientists (researchers) are more in trouble than the engineers (which i what i call an applied ML practitioner)

4

academic_wanderer t1_irfzoc0 wrote

ML umbrella now is too big, many fields were not labelled as ML some decades ago but now are under ML umbrella. My research areas are in game theory (economics), mathematical analysis, information theory, non-parametric statistic, ergodic theory, stochastic and non stochastic, etc. Now all of them are under the ML umbrella, in particular the reinforcement ML. Recently I found that my fields have some new names/ nomenclatures but similar concept: explainable AI, continual ML, online ML, lifelong ML, etc. I actually don't care the ML/AI buzzwords and the silly names they are trying to invent (as some analog terms used by different fields). I consider it is an expansion rather than advancement of ML research and it is good for multidisciplinary research, but I don't like the way some ML groups are trying to take all the light of the stage.

4

JanneJM t1_irgi4ji wrote

I am/was a computational neuroscientist. There's way more papers published in neuroscience than any one person has a hope of reading.

The best way for me to keep up was to use an rss reader. The big publishers and publication portals (such as arxiv) all have rss feeds, usually per keyword, and you follow the ones that might interest you.

Then you use the filtering available in your feed reader to get rid of any item you obviously don't care about. I was typically left with ~100 papers every morning, with about 4-5 that I'd actually download.

4

aepr88 t1_irfedo1 wrote

It always has been this way. In general, research is not about breadth, its about depth. A generalist will get you nowhere in the field.

3

111llI0__-__0Ill111 t1_irgmqxe wrote

Meanwhile in most typical business settings summary stats/t tests, linear, logistic, and random forest/xgboost is all thats used

3

issam_28 t1_ireuh15 wrote

You don't need to understand everything. You don't be generalist nowadays it's extremely demanding and time consuming.

2

DigThatData t1_irfll3d wrote

The trick I think is to stay current. If you don't get around to reading an important paper: it happens. You can't read everything. It's more important that you at least have a pulse on what is possible and what other people are doing. Look forward, not back.

2

klop2031 t1_irftior wrote

Too much in AI for one person to know it all.

2

GoofAckYoorsElf t1_irgdjma wrote

Yepp... Me. I just don't have the time and the necessary problems to solve in order to keep up to date. And for most of my daily challenges a good old linear regression or random forest is all that's needed. And if I want to get fancy, I use xgboost.

Stuff like diffusors, transformers... fascinates me. Do I understand the principle ideas behind them? Kind of, I think... Would I be able to replicate the results? Hell no! Let alone use the principles to solve my own problems (that I don't even have)...

I miss doing AI science. But at some point you've got to let go and let fresher minds do the pushing.

2

punknothing t1_irgjuzs wrote

ML is too broad to know everything. Pick a specialization.

I focus on stuff that'll help me in my day job (finance), so there are many areas of ML that are irrelevant.

When I'm curious about a problem, I google the idea and add machine learning to the search and wammo!

2

piman01 t1_irgom2j wrote

Same with... every field of everything... you don't know to know everything. It's impossible. Just work on what you want to.

2

Available_Lion_652 t1_irgkega wrote

I am on a niche with NLP, transformers, GPT, BERT, T5, but I totally ignored diffusion models(yes, they are cool, but very hard to apply only to text) I also ignore mostly all things related to CV. I ve also worked with GNN, they intersect with transformers, and I want to start study RL

1

RenewAi t1_irgvreu wrote

Yes I feel the same exact way

1