Building an Observable arXiv RAG Chatbot with LangChain, Chainlit, and Literal AI | by Tahreem Rasul | May, 2024

A tutorial on constructing a semantic paper engine utilizing RAG with LangChain, Chainlit copilot apps, and Literal AI observability.

On this information, I’ll show learn how to construct a semantic analysis paper engine utilizing Retrieval Augmented Era (RAG). I’ll make the most of LangChain as the primary framework for constructing our semantic engine, along-with OpenAI’s language mannequin and Chroma DB’s vector database. For constructing the Copilot embedded internet utility, I’ll use Chainlit’s Copilot characteristic and incorporate observability options from Literal AI. This instrument can facilitate educational analysis by making it simpler to seek out related papers. Customers may also be capable of work together immediately with the content material by asking questions in regards to the beneficial papers. Lastly, we are going to combine observability options within the utility to trace and debug calls to the LLM.

Right here is an summary of every little thing we are going to cowl on this tutorial:

• Develop a RAG pipeline with OpenAI, LangChain and Chroma DB to course of and retrieve probably the most related PDF paperwork from the arXiv API.
• Develop a Chainlit utility with a Copilot for on-line paper retrieval.
• Improve the applying with LLM observability options with Literal AI.

Code for this tutorial might be present in this GitHub repo:

Create a brand new conda setting:

conda create -n semantic_research_engine python=3.10

Activate the setting:

conda activate semantic_research_engine

Set up all required dependencies in your activated setting by operating the next command:

pip set up -r necessities.txt

Retrieval Augmented Era (RAG) is a well-liked method that lets you construct customized conversational AI purposes with your personal knowledge. The precept of RAG is pretty easy: we convert our textual knowledge into vector embeddings and insert these right into a vector database. This database is then linked to a big language mannequin (LLM). We’re constraining our LLM to get info from our personal database as a substitute of counting on prior data to reply consumer queries. Within the subsequent few steps, I’ll element how to do that for our semantic analysis paper engine. We are going to create a check script named rag_test.py to know and construct the elements for our RAG pipeline. These might be reused when constructing our Copilot built-in Chainlit utility.

Step 1

Safe an OpenAI API key by registering an account. As soon as achieved, create a .env file in your venture listing and add your OpenAI API key as follows:

OPENAI_API_KEY="your_openai_api_key"

This .env will home all of our API keys for the venture.

Step 2: Ingestion

On this step, we are going to create a database to retailer the analysis papers for a given consumer question. To do that, we first must retrieve a listing of related papers from the arXiv API for the question. We might be utilizing the ArxivLoader() package deal from LangChain because it abstracts API interactions, and retrieves the papers for additional processing. We are able to break up these papers into smaller chunks to make sure environment friendly processing and related info retrieval in a while. To do that, we are going to use the RecursiveTextSplitter() from LangChain, because it ensures semantic preservation of data whereas splitting paperwork. Subsequent, we are going to create embeddings for these chunks utilizing the sentence-transformers embeddings from HuggingFace. Lastly, we are going to ingest these break up doc embeddings right into a Chroma DB database for additional querying.

# rag_test.py 
from langchain_community.document_loaders import ArxivLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.vectorstores import Chroma
from langchain_community.embeddings import HuggingFaceEmbeddings
question = "light-weight transformer for language duties"
arxiv_docs = ArxivLoader(question=question, load_max_docs=3).load()
pdf_data = ()
for doc in arxiv_docs:
text_splitter = RecursiveCharacterTextSplitter(
chunk_size=1000,
chunk_overlap=100)
texts = text_splitter.create_documents((doc.page_content))
pdf_data.append(texts)
embeddings = HuggingFaceEmbeddings(model_name="sentence-transformers/all-MiniLM-l6-v2")
db = Chroma.from_documents(pdf_data(0), embeddings)

Step 3: Retrieval and Era

As soon as the database for a specific matter has been created, we will use this database as a retriever to reply consumer questions based mostly on the supplied context. LangChain provides a number of completely different chains for retrieval, the best being the RetrievalQA chain that we’ll use on this tutorial. We are going to set it up utilizing the from_chain_type() methodology, specifying the mannequin and the retriever. For doc integration into the LLM, we’ll use the stuff chain kind, because it stuffs all paperwork right into a single immediate.

# rag_test.py
from langchain.chains import RetrievalQA
from langchain_openai import OpenAI
from dotenv import load_dotenv
load_dotenv()
llm = OpenAI(mannequin='gpt-3.5-turbo-instruct', temperature=0)
qa = RetrievalQA.from_chain_type(llm=llm, 
chain_type="stuff", 
retriever=db.as_retriever())
query = "what number of and which benchmark datasets and duties had been 
in contrast for mild weight transformer?"
end result = qa({"question": query})

Now that we have now lined on-line retrieval from the arXiv API and the ingestion and retrieval steps for our RAG pipeline, we’re able to develop the net utility for our semantic analysis engine.

Literal AI is an observability, analysis and analytics platform for constructing production-grade LLM apps. Some key options provided by Literal AI embrace:

1. Observability: allows monitoring of LLM apps, together with conversations, middleman steps, prompts, and many others.
2. Datasets: permits creation of datasets mixing manufacturing knowledge and hand written examples.
3. On-line Evals: allows analysis of threads and execution in manufacturing utilizing completely different evaluators.
4. Immediate Playground: permits iteration, versioning, and deployment of prompts.

We are going to use the observability and immediate iteration options to judge and debug the calls made with our semantic analysis paper app.

When creating conversational AI purposes, builders must iterate by way of a number of variations of a immediate to get to the one which generates the most effective outcomes. Immediate engineering performs an important function in most LLM duties, as minor modifications can considerably alter the responses from a language mannequin. Literal AI’s immediate playground can be utilized to streamline this course of. As soon as you choose the mannequin supplier, you possibly can enter your preliminary immediate template, add any further info, and iteratively refine the prompts to seek out probably the most appropriate one. Within the subsequent few steps, we might be utilizing this playground to seek out the most effective immediate for our utility.

Step 1

Create an API key by navigating to the Literal AI Dashboard. Register an account, navigate to the initiatives web page, and create a brand new venture. Every venture comes with its distinctive API key. On the Settings tab, you’ll find your API key within the API Key part. Add it to your .env file:

LITERAL_API_KEY="your_literal_api_key"

Step 2

Within the left sidebar, click on Prompts, after which navigate to New Immediate. This could open a brand new immediate creation session.

As soon as contained in the playground, on the left sidebar, add a brand new System message within the Template part. Something in parenthesis might be added to the Variables, and handled as enter within the immediate:

You're a useful assistant. Use supplied {{context}} to reply consumer 
{{query}}. Don't use prior data. 
Reply:

In the best sidebar, you possibly can present your OpenAI API Key. Choose parameters such because the Mannequin, Temperature, and Most Size for completion to mess around with the immediate.

As soon as you’re happy with a immediate model, click on Save. You can be prompted to enter a reputation to your immediate, and an non-obligatory description. We are able to add this model to our code. In a brand new script named search_engine.py, add the next code:

#search_engine.py
from literalai import LiteralClient
from dotenv import load_dotenv
load_dotenv()
consumer = LiteralClient()
# This can fetch the champion model, you can even go a selected model
immediate = consumer.api.get_prompt(title="test_prompt")
immediate = immediate.to_langchain_chat_prompt_template()
immediate.input_variables = ("context", "query")

Literal AI lets you save completely different runs of a immediate, with a model characteristic. You may also view how every model is completely different from the earlier one. By default, the champion model is pulled. If you wish to change a model to be the champion model, you possibly can choose it within the playground, after which click on on Promote.

As soon as the above code has been added, we will view generations for particular prompts within the Literal AI Dashboard (extra on this later).

Chainlit is an open-source Python package deal designed to construct production-ready conversational AI purposes. It gives decorators for a number of occasions (chat begin, consumer message, session resume, session cease, and many others.). You may take a look at my article under for a extra thorough rationalization:

Particularly on this tutorial, we are going to deal with constructing a Software Copilot for our RAG utility utilizing Chainlit. Chainlit Copilot provides contextual steerage and automatic consumer actions inside purposes.

Embedding a copilot in your utility web site might be helpful for a number of causes. We are going to construct a easy internet interface for our semantic analysis paper engine, and combine a copilot inside it. This copilot could have a number of completely different options, however listed here are probably the most distinguished ones:

1. It is going to be embedded inside our web site’s HTML file.
2. The copilot will be capable of take actions on behalf of the consumer. Let’s say the consumer asks for on-line analysis papers on a selected matter. These might be displayed in a modal, and we will configure our copilot to do that robotically while not having consumer inputs.

Within the subsequent few steps, I’ll element learn how to create a software program copilot for our semantic analysis engine utilizing Chainlit.

Step 1

Step one includes writing logic for our chainlit utility. We are going to use two chainlit decorator capabilities for our use case: @cl.on_chat_start and @cl.on_message. We are going to add the logic from the web search and RAG pipeline to those capabilities. Just a few issues to recollect:

• @cl.on_chat_start accommodates all code required to be executed firstly of a brand new consumer session.
• @cl.on_message accommodates all code required to be executed when a consumer sends in a brand new message.

We are going to encapsulate your entire course of from receiving a analysis matter to making a database and ingesting paperwork throughout the @cl.on_chat_start decorator. Within the search_engine.py script, import all vital modules and libraries:

# search_engine.py
import chainlit as cl
from langchain_community.document_loaders import ArxivLoader
from langchain_community.vectorstores import Chroma
from langchain_community.embeddings import HuggingFaceEmbeddings
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.chains import RetrievalQA
from langchain_openai import ChatOpenAI
from dotenv import load_dotenv
load_dotenv()

Let’s now add the code for the @cl.on_chat_start decorator. We are going to make this perform asynchronous to make sure a number of duties can run concurrently.

# search_engine.py
# contd.
@cl.on_chat_start
async def retrieve_docs():
# QUERY PORTION
arxiv_query = None
# Await the consumer to ship in a subject
whereas arxiv_query is None:
arxiv_query = await cl.AskUserMessage(
content material="Please enter a subject to start!", timeout=15).ship()
question = arxiv_query('output')
# ARXIV DOCS PORTION
arxiv_docs = ArxivLoader(question=arxiv_query, load_max_docs=3).load()
# Put together arXiv outcomes for show
arxiv_papers = (f"Printed: {doc.metadata('Printed')} n "
f"Title: {doc.metadata('Title')} n "
f"Authors: {doc.metadata('Authors')} n "
f"Abstract: {doc.metadata('Abstract')(:50)}... n---n"
for doc in arxiv_docs)
await cl.Message(content material=f"{arxiv_papers}").ship()
await cl.Message(content material=f"Downloading and chunking articles for {question} "
f"This operation can take some time!").ship()
# DB PORTION
pdf_data = ()
for doc in arxiv_docs:
text_splitter = RecursiveCharacterTextSplitter(
chunk_size=1000, chunk_overlap=100)
texts = text_splitter.create_documents((doc.page_content))
pdf_data.append(texts)
llm = ChatOpenAI(mannequin='gpt-3.5-turbo',
temperature=0)
embeddings = HuggingFaceEmbeddings(
model_name="sentence-transformers/all-MiniLM-l6-v2")
db = Chroma.from_documents(pdf_data(0), embeddings)
# CHAIN PORTION
chain = RetrievalQA.from_chain_type(llm=llm,
chain_type="stuff",
retriever=db.as_retriever(),
chain_type_kwargs={
"verbose": True,
"immediate": immediate
}
)
# Let the consumer know that the pipeline is prepared
await cl.Message(content material=f"Database creation for `{question}` full. "
f"Now you can ask questions!").ship()
cl.user_session.set("chain", chain)
cl.user_session.set("db", db)

Let’s undergo the code we have now wrapped on this perform:

1. Prompting consumer question: We start by having the consumer ship in a analysis matter. This perform is not going to proceed till the consumer submits a subject.
2. On-line Search: We retrieve related papers utilizing LangChain’s wrapper for arXiv searches, and show the related fields from every entry in a readable format.
3. Ingestion: Subsequent, we chunk the articles and create embeddings for additional processing. Chunking ensures massive papers are dealt with effectively. Afterward, a Chroma database is created from processed doc chunks and embeddings.
4. Retrieval: Lastly, we arrange a RetrievalQA chain, integrating the LLM and the newly created database as a retriever. We additionally present the immediate we created earlier in our Literal AI playground.
5. Storing variables: We retailer the chain and db in variables utilizing the cl.user_session.set performance for reuse in a while.
6. Person messages: We use Chainlit’s cl.Message performance all through the perform to work together with the consumer.

Let’s now outline our @cl.on_message perform, and add the era portion of our RAG pipeline. A consumer ought to be capable of ask questions from the ingested papers, and the applying ought to present related solutions.

@cl.on_message
async def retrieve_docs(message: cl.Message):
query = message.content material
chain = cl.user_session.get("chain")
db = cl.user_session.get("db")
# Create a brand new occasion of the callback handler for every invocation
cb = consumer.langchain_callback()
variables = {"context": db.as_retriever(search_kwargs={"okay": 1}), 
"question": query}
database_results = await chain.acall(variables,
callbacks=(cb))
outcomes = (f"Query: {query} "
f"n Reply: {database_results('end result')}")
await cl.Message(outcomes).ship()

Here’s a breakdown of the code within the perform above:

1. Chain and Database Retrieval: We first retrieve the beforehand saved chain and database from the consumer session.
2. LangChain Callback Integration: To make sure we will observe our immediate and all generations that use a specific immediate model, we have to add the LangChain callback handler from Literal AI when invoking our chain. We’re creating the callback handler utilizing the langchain_callback() methodology from the LiteralClient occasion. This callback will robotically log all LangChain interactions to Literal AI.
3. Era: We outline the variables: the database because the context for retrieval and the consumer’s query because the question, additionally specifying to retrieve the highest end result (okay: 1). Lastly, we name the chain with the supplied variables and callback.

Step 2

The second step includes embedding the copilot in our utility web site. We are going to create a easy web site for demonstration. Create an index.html file and add the next code to it:

<!DOCTYPE html>
<html>
<head>
<title>Semantic Search Engine</title>
</head>
<physique>
<!-- ... -->
<script src="http://localhost:8000/copilot/index.js"></script>
<script>
window.mountChainlitWidget({
chainlitServer: "http://localhost:8000",
});
</script>
</physique>

Within the code above, we have now embedded the copilot inside our web site by pointing to the placement of the Chainlit server internet hosting our app. The window.mountChainlitWidget provides a floating button on the underside proper nook of your web site. Clicking on it’s going to open the Copilot. To make sure our Copilot is working appropriately, we have to first run our Chainlit utility. Navigate inside your venture listing and run:

chainlit run search_engine.py -w

The code above runs the applying on https://localhost:8000. Subsequent, we have to host our utility web site. Opening the index.html script inside a browser doesn’t work. As an alternative, we have to create an HTTPS testing server. You are able to do this in several methods, however one simple method is to make use of npx. npx is included with npm (Node Package deal Supervisor), which comes with Node.js. To get npx, you merely must install Node.js in your system. Navigate inside your listing and run:

npx http-server

Working the command above will serve our web site at https://localhost:8080. Navigate to the handle and it is possible for you to to see a easy internet interface with the copilot embedded.

Since we might be utilizing the @cl.on_chat_start wrapper perform to welcome customers, we will set the show_readme_as_default to false in our Chainlit config to keep away from flickering. Yow will discover your config file in your venture listing at .chainlit/config.toml.

Step 3

To execute the code solely contained in the Copilot, we will add the next:

@cl.on_message
async def retrieve_docs(message: cl.Message):
if cl.context.session.client_type == "copilot":
# code to be executed solely contained in the Copilot

Any code inside this block will solely be executed once you work together along with your utility from inside your Copilot. For instance, if you happen to run a question on the Chainlit utility interface hosted at https://localhost:8000, the code contained in the above if block is not going to be executed, because it’s anticipating the consumer kind to be the Copilot. It is a useful characteristic that you should use to distinguish between actions taken immediately within the Chainlit utility and people initiated by way of the Copilot interface. By doing so, you possibly can tailor the conduct of your utility based mostly on the context of the request, permitting for a extra dynamic and responsive consumer expertise.

Step 4

The Copilot can name capabilities in your web site. That is helpful for taking actions on behalf of the consumer, akin to opening a modal, creating a brand new doc, and many others. We are going to modify our Chainlit decorator capabilities to incorporate two new Copilot capabilities. We have to specify within the index.html file how the frontend ought to reply when Copilot capabilities in our Chainlit backend utility are activated. The particular response will range based mostly on the applying. For our semantic analysis paper engine, we’ll generate pop-up notifications on the frontend every time it is necessary to indicate related papers or database solutions in response to a consumer question.

We are going to create two Copilot capabilities in our utility:

• showArxivResults: this perform might be accountable for displaying the web outcomes pulled by the arxiv API towards a consumer question.
• showDatabaseResults: this perform might be accountable for displaying the outcomes pulled from our ingested database towards a consumer query.

First, let’s arrange the backend logic within the search_engine.py script and modify the @cl.on_chat_start perform:

@cl.on_chat_start
async def retrieve_docs():
if cl.context.session.client_type == "copilot":
# similar code as earlier than
# Set off popup for arXiv outcomes
fn_arxiv = cl.CopilotFunction(title="showArxivResults", 
args={"outcomes": "n".be part of(arxiv_papers)})
await fn_arxiv.acall()
# similar code as earlier than

Within the code above, a Copilot perform named showArxivResults is outlined and referred to as asynchronously. This perform is designed to show the formatted checklist of arXiv papers immediately within the Copilot interface. The perform signature is sort of easy: we specify the title of the perform and the arguments it’s going to ship again. We are going to use this info in our index.html file to create a popup.

Subsequent, we have to modify our @cl.on_message perform with the second Copilot perform that might be executed when a consumer asks a query based mostly on the ingested papers:

@cl.on_message
async def retrieve_docs(message: cl.Message):
if cl.context.session.client_type == "copilot":
# similar code as earlier than
# Set off popup for database outcomes
fn_db = cl.CopilotFunction(title="showDatabaseResults", 
args={"outcomes": "n".be part of(outcomes)})
await fn_db.acall()
# similar code as earlier than

Within the code above, we have now outlined the second Copilot perform named showDatabaseResults to be referred to as asynchronously. This perform is tasked with displaying the outcomes retrieved from the database within the Copilot interface. The perform signature specifies the title of the perform and the arguments it’s going to ship again.

Step 5

We are going to now edit our index.html file to incorporate the next adjustments:

1. Add the 2 Copilot capabilities.
2. Specify what would occur on our web site when both of the 2 Copilot capabilities will get triggered. We are going to create a popup to show outcomes from the applying backend.
3. Add easy styling for popups.

First, we have to add the occasion listeners for our Copilot capabilities. Within the <script> tag of your index.html file, add the next code:

<script>
// earlier code
window.addEventListener("chainlit-call-fn", (e) => {
const { title, args, callback } = e.element;
if (title === "showArxivResults") {
doc.getElementById("arxiv-result-text").innerHTML =
args.outcomes.exchange(/n/g, "<br>");
doc.getElementById("popup").type.show = "flex";
if (callback) callback();
} else if (title === "showDatabaseResults") {
doc.getElementById("database-results-text").innerHTML =
args.outcomes.exchange(/n/g, "<br>");
doc.getElementById("popup").type.show = "flex";
if (callback) callback();
}
});
</script>

Here’s a breakdown of the above code:

• Contains capabilities to indicate (showPopup()) and conceal (hidePopup()) the popup modal.
• An occasion listener is registered for the chainlit-call-fn occasion, which is triggered when a Copilot perform (showArxivResults or showDatabaseResults) is known as.
• Upon detecting an occasion, the listener checks the title of the Copilot perform referred to as. Relying on the perform title, it updates the content material of the related part throughout the popup with the outcomes supplied by the perform. It replaces newline characters (n) with HTML line breaks (<br>) to format the textual content correctly for HTML show.
• After updating the content material, the popup modal is displayed (show: "flex"), permitting the consumer to see the outcomes. The modal might be hidden utilizing the shut button, which calls the hidePopup() perform.

Subsequent, we have to outline the popup modal we have now specified above. We are able to do that by including the next code to the <physique> tag of our index.html script:

<div id="popup" class="popup">
<span class="close-btn" onclick="hidePopup()">&instances;</span>
<div class="arxiv-results-wrapper">
<h1>Arxiv Outcomes</h1>
<p id="arxiv-result-text">On-line outcomes might be displayed right here.</p>
</div>
<div class="database-results-wrapper">
<h1>Database Outcomes</h1>
<p id="database-results-text">Database outcomes might be displayed right here.</p>
</div>
</div>

Let’s additionally add some styling for our popups. Edit the <head> tag of the index.html file:

<type>
* {
box-sizing: border-box;
}
physique {
font-family: sans-serif;
}
.close-btn {
place: absolute;
high: 10px;
proper: 20px;
font-size: 24px;
cursor: pointer;
}
.popup {
show: none;
place: mounted;
high: 50%;
left: 50%;
rework: translate(-50%, -50%);
background-color: white;
padding: 20px;
box-shadow: rgba(99, 99, 99, 0.2) 0px 2px 8px 0px;
width: 40%;
flex-direction: column;
hole: 50px;
}
p {
shade: #00000099;
}
</type>

Now that we have now added our Copilot logic to our Chainlit utility, we will run each our utility and the web site. For the Copilot to work, our utility should already be operating. Open a terminal inside your venture listing, and run the next command to launch the Chainlit server:

chainlit run search.py -h

In a brand new terminal, launch the web site utilizing:

npx http-server

Integrating observability options right into a production-grade utility, akin to our Copilot-run semantic analysis engine, is often required to make sure the applying’s reliability in a manufacturing setting. We might be utilizing this with the Literal AI framework.

For any Chainlit utility, Literal AI robotically begins monitoring the applying and sends knowledge to the Literal AI platform. We already initiated the Literal AI consumer when creating our immediate within the search_engine.py script. Now, every time the consumer interacts with our utility, we are going to see the logs within the Literal AI dashboard.

Navigate to the Literal AI Dashboard, choose the venture from the left panel, after which click on on Observability. You will note logs for the next options.

Threads

A thread represents a dialog session between an assistant and a consumer. You need to be capable of see all of the conversations a consumer has had within the utility.

Increasing on a specific dialog will give key particulars, such because the time every step took, particulars of the consumer message, and a tree-based view detailing all steps. You may also add a dialog to a dataset.

Runs

A run is a sequence of steps taken by an agent or a sequence. This provides particulars of all steps taken every time a sequence or agent is executed. With this tab, we get each the enter and the output for every consumer question.

You may increase on a run, and this may give additional particulars. As soon as once more, you possibly can add this information to a dataset.

Generations

A era accommodates each the enter despatched to an LLM and its completion. This provides key particulars together with the mannequin used for a completion, the token depend, in addition to the consumer requesting the completion, if in case you have configured a number of consumer classes.

We are able to observe generations and threads towards every immediate created and used within the utility code since we added LangChain integrations. Due to this fact, every time the chain is invoked for a consumer question, logs are added towards it within the Literal AI dashboard. That is useful to see which prompts had been accountable for a specific era, and evaluate efficiency for various variations.

On this tutorial, I demonstrated learn how to create a semantic analysis paper engine utilizing RAG options with LangChain, OpenAI, and ChromaDB. Moreover, I confirmed learn how to develop an online app for this engine, integrating Copilot and observability options from Literal AI. Incorporating analysis and observability is mostly required for making certain optimum efficiency in real-world language mannequin purposes. Moreover, the Copilot might be an especially helpful characteristic for various software program purposes, and this tutorial could be a good start line to know learn how to set it up to your utility.

Yow will discover the code from this tutorial on my GitHub. Should you discovered this tutorial useful, take into account supporting by giving it fifty claps. You may follow along as I share working demos, explanations and funky aspect initiatives on issues within the AI area. Come say hello on LinkedIn and X! I share guides, code snippets and different helpful content material there. 👋