Nerf was a favorite brand of mine growing up and I am glad that it has a second life as a synonym for ‘hitting the easy button’. Setting up an LLM in deployment without significant expertise as a cloud architect is a bit like throwing an NFL football as a child: awkward, usually unsuccessful, and occasionally dangerous. The dangerous part of deploying LLMs is the cost-effective management of very expensive compute resources. Anyscale Nerfs the entire process by including a basic chatbot program in Python and using Ray behind the scenes. Read on to learn more about my experience with Anyscale and the feature they created to enhance throughput when loading models and reduce latency for LLM performance in deployment!
Since August I have been using an Anyscale Endpoint to query Llama-2 70b through a Terminal window in VS Code. I have used the service to generate blog content and it is very impressive. I am currently working on several basic containerized applications that expose the API. I am taking the CKA exam later this month and will push the new apps after that. This blog post describes the challenge of reducing latency when serving Large Language Models (LLMs) in production. I have completed two Nvidia training courses using Generative AI, have used Hugging Face Transformers, and have an active Nvidia developer account. The Riva ecosystem is powerful and is not easy to deploy. Anyscale has streamlined the integration process with Hugging Face to create a very comfortable experience for developers.
Anyscale has abstracted away much of the complexity of loading a model via the Anyscale Model Loader which is described in the second half of the blog post:
At Anyscale, we have built Anyscale Model Loader (AML) to speed up the loading models more efficiently. There is a downloader, part of AML, pulling data from S3 concurrently with more than 250 threads. Each thread holds an 8MB buffer. Once the fetching is finished, it’ll write it to the GPU buffer directly and then start to fetch the next chunk. It’s fully integrated with vLLM for the loading and being used in Anyscale Endpoint.
I recommend reading the entire blog post to understand the challenge of reducing latency when deploying LLMs. Try Anyscale Endpoints to deploy Llama-2 70b or another model of your choice!
https://www.anyscale.com/blog/loading-llama-2-70b-20x-faster-with-anyscale-endpoints?