RAG At Scale: The Problems You Will Encounter And How To Prevent (or Fix) Them

Το περιεχόμενο παρέχεται από το Nicolay Gerold. Όλο το περιεχόμενο podcast, συμπεριλαμβανομένων των επεισοδίων, των γραφικών και των περιγραφών podcast, μεταφορτώνεται και παρέχεται απευθείας από τον Nicolay Gerold ή τον συνεργάτη της πλατφόρμας podcast. Εάν πιστεύετε ότι κάποιος χρησιμοποιεί το έργο σας που προστατεύεται από πνευματικά δικαιώματα χωρίς την άδειά σας, μπορείτε να ακολουθήσετε τη διαδικασία που περιγράφεται εδώ https://el.player.fm/legal.

How AI Is Built « »
RAG at Scale: The problems you will encounter and how to prevent (or fix) them | S2 E4

2M ago 50:09

Μοίρασέ το

MP3•Αρχική οθόνη επεισοδίου

Hey! Welcome back.

Today we look at how we can get our RAG system ready for scale.

We discuss common problems and their solutions, when you introduce more users and more requests to your system.

For this we are joined by Nirant Kasliwal, the author of fastembed.

Nirant shares practical insights on metadata extraction, evaluation strategies, and emerging technologies like Colipali. This episode is a must-listen for anyone looking to level up their RAG implementations.

"Naive RAG has a lot of problems on the retrieval end and then there's a lot of problems on how LLMs look at these data points as well."

"The first 30 to 50% of gains are relatively quick. The rest 50% takes forever."

"You do not want to give the same answer about company's history to the co-founding CEO and the intern who has just joined."

"Embedding similarity is the signal on which you want to build your entire search is just not quite complete."

Key insights:

Naive RAG often fails due to limitations of embeddings and LLMs' sensitivity to input ordering.
Query profiling and expansion:
- Use clustering and tools like latent Scope to identify problematic query types
- Expand queries offline and use parallel searches for better results
Metadata extraction:
- Extract temporal, entity, and other relevant information from queries
- Use LLMs for extraction, with checks against libraries like Stanford NLP
User personalization:
- Include user role, access privileges, and conversation history
- Adapt responses based on user expertise and readability scores
Evaluation and improvement:
- Create synthetic datasets and use real user feedback
- Employ tools like DSPY for prompt engineering
Advanced techniques:
- Query routing based on type and urgency
- Use smaller models (1-3B parameters) for easier iteration and error spotting
- Implement error handling and cross-validation for extracted metadata

Nirant Kasliwal:

Nicolay Gerold:

query understanding, AI-powered search, Lambda Mart, e-commerce ranking, networking, experts, recommendation, search

33 επεισόδια

#Tech #Nicolay Gerold #Technology #Llm #Machine Learning #Data Engineering