theLastNenUser t1_j9uwhcd wrote on February 24, 2023 at 7:01 PM Reply to comment by Desticheq in [P] What are the latest "out of the box solutions" for deploying the very large LLMs as API endpoints? by johnhopiler You will have to message them if you want to use the larger GPU boxes, and the autoscaling isn’t great for larger models. The customizability of the “handler.py” file is nice though Permalink Parent 2
theLastNenUser t1_j9uwhcd wrote
Reply to comment by Desticheq in [P] What are the latest "out of the box solutions" for deploying the very large LLMs as API endpoints? by johnhopiler
You will have to message them if you want to use the larger GPU boxes, and the autoscaling isn’t great for larger models. The customizability of the “handler.py” file is nice though