Vectorize Docs for RAG Chatbot

[patriciaflutterflow]

Link to RAG Chatbot PRD

There are 2 tables in BigQuery associated with documentation for the RAG Chatbot:

1. doc_text (includes doc id + doc text)
2. doc_text_vector (includes doc id + chunk id + vectorized chunk + chunk text)
   The code here ultimately processes each documentation page to write to the tables in BigQuery.

What happens:

• Github action run everytime there is a change in a markdown file in docs (vectorize_docs.yaml), function is process_single_file in vectorize.py
• the markdown file is chunked and each chunk is vectorized
• these changes are added to bigquery
• there is also a function that does a backfill for all existing docs

ToDo: right now, I am getting a 429 error quota exceeded after vectorizing sometimes. Check to see what quota limits are and how to get around it.

[bolt-new-by-stackblitz]

Run & review this pull request in StackBlitz Codeflow.

[PoojaB26]

Is there a ticket to this or maybe an explanation of what it does? @patriciaflutterflow

[patriciaflutterflow]

Is there a ticket to this or maybe an explanation of what it does? @patriciaflutterflow

Hello @PoojaB26 ! Yes added a brief overview at the top