"Scaling RAG Applications to serve millions of users", Kevin Goedecke

•

0 likes•147 views

How we managed to grow and scale a RAG application from zero to thousands of users in 7 months. Lessons from technical challenges around managing high load for LLMs, RAGs and Vector databases.

More Related Content

"Scaling RAG Applications to serve millions of users", Kevin Goedecke

1. Scaling RAG applications to serve millions of users Lessons learned scaling LLMs, Vector databases and more… Powered by

2. What we’ll cover today… 📝 Agenda - Who is SlideSpeak? What do we do? - Our Growth so far - Mistakes we’ve made - Architecture Overview - Challenges of scaling RAG applications - Challenges of scaling Vector databases - What we’re currently busy with… - Q&A

3. Create presentations from Word, PDF or Excel files Create presentations for any topic Some of our key features: SlideSpeak is an AI-powered platform to create and automate presentations What we do

4. We save people hours of work by speeding up manual presentation workflows

5. OUR GROWTH SO FAR We’ve launched 6 months ago, here’s what we’ve achieved. >2m Files upload >2m LLM Tokens consumed per minute >1000 LLM calls per minute 250k MAUs

6. User Vector DB Context LLM Response Query Inference Retrieval Prompt Quick recap on RAG

7. Using LLMs for everything Storing vector data forever No monitoring Downtimes cost us thousands of $$$ Mistakes we’ve made… so far… �� Storing vectors is expensive, like really expensive… If not absolutely necessary use avoid using LLMs

8. Architecture

9. Prompts, JSON Mode, Regression Scaling Vector Databases Scaling LLM providers Rate limits, Downtimes, … Challenges scaling RAG Applications �� Slow queries, index building, … LLM specific prompts, Pydantic being unreliable, …

10. - Rate limits are not as high as they seem - Difficult to balance RPM and TPM - 40 page document has on avg. 24k tokens, with 2m limit thats 83 documents per minute or 1.5 per second OpenAI Rate Limits (6/10/24) Scaling LLM providers The problem

11. - We’ve migrated to Azure OpenAI - Not because the rate limits are higher, but you can load balance 🤯 https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models Scaling LLM providers The solution

12. Source: https://techcommunity.microsoft.com/t5/fasttrack-for-azure/smart-load-balancing-for-openai-end points-and-azure-api/ba-p/3991616 Scaling LLM providers Load Balancing in Azure

13. Scaling LLM providers The solution

14. - Find similar information in a lot of data - Feed that to the LLM as context Why do we need Vector Databases?

15. 5 Problems when scaling Postgres with PGVector - 🦥 Slow queries - 🧠 Memory intense - 🔍 Challenge of combining vector search and metadata search (also called hybrid search) needs careful query optimization Challenges of scaling Vector Databases

16. Challenges of scaling Vector Databases A Vector is an array of 4-byte floating point numbers. Number of Vectors Total Size 7 Thousand 7,000 × 1536 × 4 = 43 Megabyte 1 Million 1,000,000 × 1536 × 4 = 5.7 Gigabyte 10 Millions 10,000,000 × 1536 × 4 = 57 Gigabyte 1 Billion 1000,000,000 × 1536 × 4 = 5.5 Terabyte

17. Build an index over the vector data Challenges of scaling Vector Databases HNSW: Efficient nearest neighbor search algorithm (ANN) for high-dimensional data. ef_construction: Defines how many similarity candidates to look for m: Defines how many of the closest neighbors to pick from the ef_construction list ** This might be more tricky if you use hybrid search (metadata + vector data)

18. Define what could be a good partition value for you (date, filetype, category, …) Use partitioning What we’ve done to scale PGVector

19. General tips when working with PGVector Make sure to delete unnecessary vector data 01 Never use the same Postgres database for Vectors and other data 02

20. If nothing helps… Scale horizontally What we’ve done to scale PGVector

21. What we’re currently busy with… System Extension Azure AI Postgres Extension RAG Evaluation – Implementing robust evaluation testing methods – Exploring Ragas for advanced system performance ✅ �� – Enabling direct creation of embeddings within Azure – Extend system to cover images and other non-textual data Here’s what currently keeps us up at night…

22. Questions? And now it’s your turn…

"Scaling RAG Applications to serve millions of users", Kevin Goedecke

Related slideshows

More Related Content

"Scaling RAG Applications to serve millions of users", Kevin Goedecke