Automate Pinecone Day by day Upsert Task with Celery and Slack monitoring
It’s been some time since my last LLM post and I’m excited to share that my prototype has been successfully productionized as Outside’s first LLM-powered chatbot, Scout. In case you are an Outside+ member, you may test it out over at https://scout.outsideonline.com/.
This journey began as my weekend curiosity project back in March 2023. I had the thought to construct a Q&A chatbot using OpenAI’s LLMs and Outside’s content as a knowledge base. Later I shared my prototype at our internal product demo day and I used to be thrilled by the interest it managed to spark. Scout quickly became an official project. On November twenty eighth 2023, we launched Scout to limited Outside+ members. Fast forward to today, April twelfth, 2024, over 28.3k unique users have already utilized this Outdoor Companion AI tool.
I couldn’t be more grateful for this moonstruck experience and I’ve been planning to jot down a mini-series to share some behind-the-scenes insights into what it takes to bring LLM & RAG powered apps to life. Up to now I’ve planned to cover the next three parts:
- 🦦 Part 1: Automate Pinecone Day by day Upserts with Celery and Slack monitoring
- 🦦 Part 2: Constructing an LLM Websocket API in Django with Postman Testing
- 🦦 Part 3: Monitoring LLM Apps with Datadog: synthetic tests, OpenAI, and Pinecone usage tracking
This post will dive into Part 1, organising scheduled tasks with Celery Beat to mechanically upsert embeddings into the Pinecone vector database. And we’ll arrange slack updates for simple monitoring. Let’s start!
LLMs typically have training data cut off date, the present gpt-4-turbo was cut off at 2023-Dec (to my writing day -2024-April). The promise of using RAG is that we will equip LLMs with more fresh and domain specific data to scale back hallucinations and improve user experience. Thus the query: how can we keep the knowledge base fresh and up so far? The reply is — using Celery and Celery Beat to schedule a periodical task (each day or weekly) to embed newly published…