This is an infamous Software Engineering smell/tech-debt issue. Unfortunately, you hitched your wagon to the wrong horse - and this is why framework choice is a hugely important matter in all Engineering teams. And doing your research before commiting to structing entire projects around a framework may seem like a waste of time but it can prevent major headaches like this.

In reality, there is no shortcut to rewriting / disentangling your project from AllenNLP to more conventional NLP (Huggingface, Spacy) which are all but guaranteed to be safe for long term use.

It's now a matter of convincing Product / the business that you need to commit a lot of time to an Epic that won't deliver direct value.

​

>how to ensure that everything is reproduced

Unit tests, integration tests, logging & monitoring.

And of course comparing ML metrics for each model that was produced in AllenNLP versus the replacement models produced with Spacy/Huggingface/native-pytorch.

​

> and how to not want to pull out all your hair in frustration while doing it

Sorry mate, but there's no avoiding this one. Good luck.