feat(docs): add AI chatbot

[MFarabi619]

What feature should we improve? Please describe.
Having documentation is great, and the latest trends in AI/ML have proven that LLM-powered searches are the future. As our documentation grows, it will take more and more time to browse. Integrating an LLM into our codebase and docs can greatly accelerate our learning and productivity, and also reduce the chances of duplicate, conflicting information.

Describe the solution you'd like
I came across NextUI's docs AI integration and was blown away. Something like this is exactly what I'd like to implement for us.

Describe alternatives you've considered
I use doc-gen in tandem with Node scripts as much as possible to make documentation keep itself up-to-date, however it doesn't really still solve the problem of actually having to browse the docs to find the information you need.

The search functionality is a step in the right direction, but slows down as content grows. You get to a point where there are too many results to look through, kind of like what it's like to search on google. It takes a bit of time before you start to memorize the locations in the documentation that contain the information you need.

Integrating AI would serve as a straight arrow for users to help them find exactly what they need, showing them where they can find it, and phrasing answers in a way that is suitable for them to read as an individual.

Additional context
Having quality docs is rare, and the fact that we have it already puts us ahead very much ahead when it comes to building hackathon platforms. I'd like to take it all the way by making the experience outstanding instead of just great. It will also serve as a source of inspiration for hackers attending our hackathon, showcasing our commitment to using the latest cutting-edge tools to provide both ourselves and them with the best possible hackathon experience.

[JeremyFriesenGitHub]

to follow up on this @MFarabi619, next ui doesn't power the chatbot themselves, but rather is powered by kapa.ai. Kapa is not natively open-source, but they do recognize and have an open-source tier here. This is also the link to their docs as well. I will be looking into alternatives to this and will make an ADR about this to see which is the best bot for our docs site. From my understanding, kappa is very well supported (docker uses it for their docs as an example) and would be a solid solution to integrate into our project. More to come on this...

[MFarabi619]

https://discord.com/channels/1193362882302320751/1193362882302320754/1269809429004685355

[JamesGough24]

Hey guys I'm James, I'm a second-year CS student at Carleton and I'm really interested in helping you guys create this cool feature. Hasith mentioned what you were trying to do, so I’ve been thinking about a potential solution for the last few days and here are my thoughts:

Kapa.ai’s services are very trusted and would be tailormade for the task at hand, so that is probably the best option. Not sure how open they would be to offering it to you open source, but always worth a try.

If that is not an option, it would be possible to create the chatbot just regularly using open source tools and free tier solutions out there.

RAG systems are becoming more and more popular, providing an efficient way of utilizing an external knowledge base to query an LLM. This knowledge base can be made up of scraping the pages of the docs site, cutting it up into chunks (separating the content by section, for example), generating embeddings for each chunk, and storing them in a vector database, alongside some metadata regarding where in the website that text was found (would make the prompt more accurate). At first I thought the best route for this would be FAISS (Facebook AI Similarity Search) or ChromaDB but then you’d have to host that locally so a better option would be Pinecone, a vector database service that manages all that for you. They have a free tier that I believe would provide enough storage and queries given the size that the docs site will be.

With that in place, a prompt entered by the docs site user can be analyzed against the chunks taken from the docs site to find the most relevant information (Similarity search). The LLM of choice would then be given the most relevant chunks, along with the original prompt (that is slightly optimized for best results) to form the response. To run this whole backend, you could use Google Cloud Run. Its free tier allows for lots of compute time and many requests so would be very scalable as the site grows in popularity (or perhaps you are in the process of running a hackathon and traffic is greatly increased). To connect all of this, FastAPI would be a good choice. It’s fast, it’s widely used in the industry, it can connect to the frontend to receive the prompt and then integrate the vector db and LLM query.

All of that’s great but the most problems will arise from the LLM of choice. Really, there’s two choices; OpenAI or Llama. They both have APIs where you can provide prompts and knowledge base information, however they have their own drawbacks. Llama is fully open source which is fantastic, however it requires a lot of compute resources. Even if you were to get it working after lots of optimization, it would be extremely slow, and this is not the UX that you want to provide. OpenAI on the other hand is, in my opinion, the much better option. The only thing is that it’s free only until a certain point. Queries to the chatbot produce tokens and, after doing some research, it seems like using the normal RAG process and prompt length would get you about 2000 queries/month before breaching the token limit. In other words, 2000 questions from users a month, or 67 questions a day. Not bad but also not the greatest (especially during a hackathon).

To reduce the number of tokens used, there’s a few things you can do:

1. Store cache of prompts in a simple db, something small like SQLite or even a json file somewhere would do (really would not take up that much space), so that if users ask a similar question, it returns the previous response it made, rather than having to call the OpenAI API again. This would be very effective.
2. Only send 3-5 most relevant chunks to the llm, would still get enough context while also not taking up as many tokens
3. Optimize chunk sizes in the first place, but the docs site doesn’t have massive paragraphs of text anyways so this is pretty much already optimized

They’re pretty simple fixes in reality, and would greatly increase the number of prompts the chatbot can receive each month.

Let me know what you think, if you need clarification on something, or anything of the sort! Again, I would love to help you guys implement it if that’s something you’re willing to offer.

[MFarabi619]

Hey @JamesGough24, we're super tied up with launch right now. Rest assured that this feature is very high on my list of priorities. We'll be in touch :)