Are you using these as in a system? For just experimenting around, ec2 is good option. But you would either need to install right drivers or use latest deep learning ami. Another option could be using a custom docker setup on sagemaker. I like that setup for inference as it’s super easy to deploy and separates model from inference code. Though it’s costlier and would be available through sagemaker runtime.

Third would be whole over engineering via setting up your own cluster service.

In general if you want to deploy multiple llm quickly go for sagemaker

On a side note sagemaker was not supporting shm-size so might not work for large lm

Use a spot instance. If you testing it out you wallet will thank you later. Look at my previous post on here about running stuff in the cloud before you do it.

[removed]

maybe, Colab Pro+?

Using vast.ai for running flan-xl, works pretty well. Haven't tested on LLama scale.

or you could use any gpu workload specific provider https://www.paperspace.com/

Good luck getting a EC2 with a single A100, last time I checked, AWS only offered instances with 8 of them at a high price.

Check out Banana. They seem to do exactly that with "serverless" A100.

What config did you use?

Related, there's a talk on Thursday about running LLMs in production. I think the hosts have deployed LLMs in prod so they should have good advice

Maybe check datacrunch.io they have a good offering for cloud GPU.

I just got instance at 8X RTX A5000 for a couple of bucks per hour. on https://vast.ai

I must say LLaMA 65B is a bit underwhelming...

Underwhelming how?

if latency is not a critical requirement, you can try serverless GPU cloud like banana.dev, pipeline.ai . These platform provide an easy to use template for deploying LLM.