sgt102

sgt102 OP t1_iwhfqco wrote

Reply to comment by BATTLECATHOTS in [D] I Wrote a book "Managing Machine Learning Projects" by sgt102

Party like 1989... I think things have changed - my teams need data pipelines and reproducible test results; and they're doing things like evaluating performance using MABs... CRISP doesn't help so much with that... Also actually building a system, not only extracting a model from a table...

Do you see CRISP as sufficient now?

2

sgt102 OP t1_iwhfc9e wrote

Reply to comment by globalminima in [D] I Wrote a book "Managing Machine Learning Projects" by sgt102

Chapter 9 addresses (to some extent) logging and monitoring, and goverance - which is a lot to do with how the model should be managed in life....

I've worked in projects where the model was ungoverned and went wrong and no one noticed for a long time... and that caused damage. I also got called in to sort out a project where the team retrained the model every week... and every week they overfitted it on new data. I think knowing what the models should do, being able to say that they are doing that and then having a clear way of deciding what to do if they aren't (ie. someone in charge) is the base of maintaining them... what's your pov though?

2

sgt102 OP t1_iwdwyr5 wrote

Reply to comment by maybe_yeah in [D] I Wrote a book "Managing Machine Learning Projects" by sgt102

The target audience is people who are being asked to lead an ML project for the first time - or who aspire to do so. The book doesn't try to teach the implementation details of modelling - mostly because there are many texts that do that very well already, far better than I could. So there are no code examples.

3

sgt102 OP t1_iwdedyf wrote

Reply to comment by Peantoo in [D] I Wrote a book "Managing Machine Learning Projects" by sgt102

Great question; very thought provoking!

I don't go through to in life CI/CD scenarios in the book, but I do look at running MAB's and A/B testing to understand the relative performance of models in live, and also write about the need for model monitoring and governance supporting the prod deployment.

Basically the book mostly ends with getting it into prod - but with the emphasis on getting it into prod with the right framework around it that it can be kept alive in prod.

−6