Skip to content

Craft local LLM labeling pipeline with Ollama, Langchain & LabelStudio

In this article we will explore how to build an end to end LLM labeling pipeline using Ollama, Langchain and Label Studio. We will see how to setting up the environment, configuring API connections, and building a labeling flow that integrates LLM responses and rating feedback.

First let's take a look at the official documentation in order to have a big picture of what we will building 😎

Setting Up the Development Environment

To start, you’ll need Python and several libraries to handle the LLM interactions, logging, and Label Studio for managing labeling tasks.

Install Python if not already installed. Python 3.10+ is recommended. You can download it from Python’s official site.

Set up a virtual environment and install packages

python3 -m venv ollama_labeling_env
source ollama_labeling_env/bin/activate  # Activate the environment on macOS/Linux
ollama_labeling_env\Scripts\activate     # Activate on Windows

Then you can install the required packages like usual :

pip install langchain_openai langchain_core labelstudio-client logging

Install and configure Label Studio

Follow the official instruction. I prefer the docker installation but you are free to choose what's suits you the best 🤗

Note the URL (usually http://localhost:8080 if it's not used) and your API key for Label Studio, which will be used in the code.

Set Up Ollama for LLM Compatibility

You can take a look at the langchain documentation on the subject.

We will by passing by an other approach called OpenAI server compatibility since Ollama provides compatibility with OpenAI models through an API.

Follow these steps to set it up:

  1. Download and install Ollama following the official instructions.
  2. Start the Ollama service locally, ensuring it runs on http://localhost:11434/v1 to interact with the LLM model.

Chill and automate with makefile

In his part let's write a makefile in order to automate the previous tasks, because it's a lot of commands and we don't want to run theses all the time. 😎

If you do not know much about makefile you can check the article in the linux section here 

# Makefile for automating build, serve, test the project

#Variables
COMPOSE_FILE = docker-compose.yml
SERVICES = app ollama

ifneq ($(OLLAMA_WEBUI),)
    SERVICES += ollama-webui
endif

.PHONY: clean dev-run dev-test pull-model check-postgres postgres-migrate help

# check if Docker PostgreSQL is running; if not, start it
check-postgres:
    @echo "Checking if PostgreSQL container is running..."
    @if ! docker ps | grep -q postgres; then \
        echo "PostgreSQL is not running. Starting it..."; \
        docker-compose -f docker-compose-postgres.yml up --build -d; \
    else \
        echo "PostgreSQL is already running."; \
    fi


#clean up images, containers, volumes, and networks created by docker-compose
clean:
    docker-compose -f $(COMPOSE_FILE) down --rmi all --volumes --remove-orphans

# apply alembic migration 
postgres-migrate:
    @if [ -z "$(MESSAGE)" ]; then \
        echo "Please provide a migration message using 'make migrate MESSAGE=\"your message\"'"; \
        exit 1; \
    else \
        alembic revision --autogenerate -m "$(MESSAGE)"; \
        alembic upgrade head; \
    fi

#local install in virtualenv
dev-install: 
    @if [ ! -d "env" ]; then \
        echo "Creating Python virtual environment..."; \
        python3 -m venv env; \
    fi
    @echo "Activating virtual environment and installing dependencies...";
    @. env/bin/activate && pip install -r requirements.txt

#test Ollama and run the uvicorn server
dev-run:
    @OUTPUT=$$(curl -s -X POST http://localhost:11434/api/generate -H 'Content-Type: application/json' \
        -d '{"model": "llama3.2", "prompt": "hello there"}'); \
    if [ $$? -eq 0 ]; then \
        echo "Ollama is running."; \
        echo "Response from Ollama:"; \
        echo "$$OUTPUT"; \
    else \
        echo "Ollama is not running. Please start Ollama before running the server."; \
        exit 1; \
    fi
    uvicorn app.main:app --host 0.0.0.0 --port 8001 --reload

#run label studio
#TODO: fix @if ! docker ps | grep heartexlabs/label-studio; 
label-studio:
    @echo "Checking if Label Studio Container is running..."
    @if ! docker ps | grep heartexlabs/label-studio; then \
        echo "Label Studio Container is not running. Starting it..."; \
        docker run -d -p 8080:8080 -v $(pwd)/tmp:/label-studio/data heartexlabs/label-studio:latest \
    else \
        echo "Label Studio Container is already running."; \
    fi

#pull some Ollama model
pull-model:
    @echo "Running Ollama model: $(MODEL_NAME)"
    @OUTPUT=$$(ollama pull $(MODEL_NAME)); \
    if [ $$? -eq 0 ]; then \
        echo "Output from 'ollama run $(MODEL_NAME)':"; \
        echo "$$OUTPUT"; \
    else \
        echo "Error running Ollama model '$(MODEL_NAME)'."; \
        exit 1; \
    fi

#help command to list all targets
help:
    @echo "Available targets:"
    @echo "  clean         - Remove images, containers, volumes, and orphans"
    @echo "  dev-install   - Install package inside a virtual environment
    @echo "  dev-run       - Test if Ollama is running and run the uvicorn server"
    @echo "  dev-test      - Test if FastAPI is running and run the Ollama model server"
    @echo "  label-studio   - Run label studio docker container"
    @echo "  pull-model    - Pull an Ollama model (specify MODEL_NAME=)"
    @echo "  postgres-migrate    - Run alembic migration in case of database change  (specify MESSAGE=)"
    @echo "  help          - Display this help message"
    @echo ""
    @echo "Note: To include 'ollama-webui' service, run 'make up OLLAMA_WEBUI=true'"

Add Postgres database to save results if needed

In this (optionnal) part let's see how to add a database and save our local results inside a postgres container. FIrst let's run a postgres container with this docker-compose-postgres.yml file here : 

version: '3.8'

services:
  db:
    image: postgres:13
    container_name: postgres_db
    restart: always
    environment:
      POSTGRES_USER: yourusername
      POSTGRES_PASSWORD: yourpassword
      POSTGRES_DB: yourdatabase
    volumes:
      - db_data:/var/lib/postgresql/data
      - ./db_dumps:/db_dumps  # To store database dumps
    ports:
      - "5432:5432"

volumes:
  db_data:

Integrate postgres into our script

Now let`s add it to our python script, first ensure the directories exist:

mkdir db_dumps

This directory will store your database dumps.

Then, install SQLAlchemy and psycopg2:

pip install SQLAlchemy psycopg2-binary

Set Up Database Connection

Create a database.py file 

from sqlalchemy import create_engine
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker
import os

DATABASE_URL = os.getenv(
    "DATABASE_URL",
    "postgresql://yourusername:yourpassword@localhost:5432/yourdatabase"
)

engine = create_engine(DATABASE_URL)
SessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=engine)
Base = declarative_base()

Some explanations of the code above :

  • DATABASE_URL: Reads from environment variable or defaults to local connection.
  • Engine: Creates a database engine.
  • SessionLocal: Creates a session factory.
  • Base: Base class for declarative models.

Create Models

Now you can create a models.py file for specify to postgres the shape of what we will be giving as database input:

from sqlalchemy import Column, Integer, String, Text
from .database import Base

class Prompt(Base):
    __tablename__ = 'prompts'
    id = Column(Integer, primary_key=True, index=True)
    language = Column(String)
    prompt = Column(Text)

class GeneratedContent(Base):
    __tablename__ = 'generated_content'
    id = Column(Integer, primary_key=True, index=True)
    language = Column(String)
    content = Column(Text)
  • Prompts: Stores our prompts.
  • GeneratedContent: Stores the generated content from the script.

Create Database Tables

In your main application file (e.g., main.py or an initialization script), add:

# main.py or app/__init__.py

from .database import engine
from . import models

models.Base.metadata.create_all(bind=engine)

This will create the tables in the database when the application starts.

Dependency for Database Session

In app/dependencies.py, create a dependency to get the database session, it will be practical when we will be playing with the database acress files :

from .database import SessionLocal
from fastapi import Depends

def get_db():
    db = SessionLocal()
    try:
        yield db
    finally:
        db.close()

Modify our script to use postgres

In your main.py or whatever script named file, update the imports:

from sqlalchemy.orm import Session
from .database import SessionLocal, engine
from .models import Prompt, GeneratedContent
from .dependencies import get_db

Then you can update your script to save and retrieve prompts and generated content from the database:

# ... your existing code here ... 

# Check if the transcript already exists in the database
prompt = db.query(Prompt).filter_by(prompt=prompt, language=language).first()
if prompt:
    logging.info(f"prompt found in database")
    return {
        "language": language,
        "prompt": prompt
    }

# Fetch the prompts 
try:
    # ... fetch your prompt to extract text ...
    prompt_text = ' '.join([t['text'] for t in prompts])

    # Save the transcript to the database
    new_prompt = Pormpt(
        prompt=video_id,
        language=language,
    )
    db.add(new_prompt)
    db.commit()
    db.refresh(new_prompt)

    logging.info(f"Prompt saved to database")

    return {
        "prompt": prompt,
        "language": language,
    }

except Exception as e:
    # ... (Handle exceptions as before)
    raise HTTPException(status_code=500, detail=f"An unexpected error occurred: {str(e)}")

Write a function to automate the dump

Let's create a utils.py file, with a function to dump the postgres database at a certain time, let's say at each recording.

import subprocess
import datetime
import os

def dump_database():
    dump_dir = './db_dumps'
    if not os.path.exists(dump_dir):
        os.makedirs(dump_dir)

    timestamp = datetime.datetime.now().strftime('%Y%m%d_%H%M%S')
    dump_file = f'dump_{timestamp}.sql'
    dump_path = os.path.join(dump_dir, dump_file)

    command = [
        'docker', 'exec', 'postgres_db',
        'pg_dump', '-U', 'yourusername', 'yourdatabase'
    ]

    with open(dump_path, 'w') as f:
        subprocess.run(command, stdout=f)

    logging.info(f"Database dumped to {dump_path}")

Now we can use our function easily just like this : 

# After inserting transcript
db.commit()
db.refresh(new_transcript)

# Dump the database
dump_database()
Verify database entries

Now you can connect to the PostgreSQL database to verify that entries are being saved 🥸

Connect to the Database:

docker exec -it postgres_db psql -U yourusername -d yourdatabase

Query the Tables:

SELECT * FROM prompts;
SELECT * FROM generated_content;

And finally you can verify that database dump files are being created in the db_dumps directory after each insertion.

Handling postgres migrations

I'll advise you using Alembic for database migrations. This will help manage schema changes over time.

Ok I know the documentation is not very good looking but give it a try, or at least use the pre maid makefile 😂

Building the LLM labeling pipeline

The pipeline is designed to initialize an Ollama-compatible LLM model, communicate with Label Studio, and handle logging and callback functions for labeling. 

import langchain 
import logging
langchain.debug = True
from langchain_community.callbacks.labelstudio_callback import (
    LabelStudioCallbackHandler,
)

logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')

logging.info("Script started")

# Initialize the Ollama LLM mimiting OpenAI 
#https://ollama.com/blog/openai-compatibility
logging.info("Initializing ChatOllama LLM with OpenAI connector")

from langchain_openai import ChatOpenAI
from langchain_core.messages import HumanMessage, SystemMessage

project_config = """
<View>
<View className="root">
     <Paragraphs name="dialogue"
               value="$prompt"
               layout="dialogue"
               textKey="content"
               nameKey="role"
               granularity="sentence"/>
  <Header value="Final response:"/>
    <TextArea name="response" toName="dialogue"
              maxSubmissions="1" editable="true"
              required="true"/>
</View>
<Header value="Rate the response:"/>
<Rating name="rating" toName="dialogue"/>
</View>
"""

handler = LabelStudioCallbackHandler(
            api_key='38e921a5ca7b6cb8ee2074012a5c5fdffde258f0',
            url='http://localhost:8080',
            project_name='Test Project',
            project_config=project_config,
            mode='chat',
)

chat_llm = ChatOpenAI(
    api_key="ollama",
    model="llama3.2",
    base_url="http://localhost:11434/v1",
    callbacks=[handler],
)

Code walkthrough

Let’s break down the code step-by-step.

Define the project config

In order to not use the label studio GUI we can pass the XML template file to the handler

project_config = """
<View>
    <View className="root">
        <Paragraphs name="dialogue"
                value="$prompt"
                layout="dialogue"
                textKey="content"
                nameKey="role"
                granularity="sentence"/>
    <Header value="Final response:"/>
    <TextArea name="response" toName="dialogue"
                maxSubmissions="1" editable="true"
                required="true"/>
    </View>
    <Header value="Rate the response:"/>
    <Rating name="rating" toName="dialogue"/>
</View>
"""
handler = LabelStudioCallbackHandler(
            api_key='YOUR_LABEL_STUDIO_API_KEY',
            url='http://localhost:8080',
            project_name='Test Project',
            project_config=project_config,
            mode='chat',
)
Define and send prompts to labelstudio
sys_prompt = """You are a veteran software python engineer, responding to user input and context summary below, create the detailed courses..."""

llm_results = chat_llm.invoke(
    [
        SystemMessage(content=sys_prompt),
        HumanMessage(content="Python API courses"),
    ]
)
  • Prompt Design: The sys_prompt string instructs the LLM to provide detailed responses, in this case, for a "Python API courses" outline.
  • Invoke Method: invoke sends SystemMessage and HumanMessage to the LLM, where:
  • SystemMessage sets context.
  • HumanMessage provides the specific user query.

You will have the following dashboard reacting to your code like below :

Add SequentialChain into the pipeline

Now let's play with the SequentialChain langchain class in order to add a better frame to our LLM and prevent it to hallucinate 😅

If you are not comforable with langchain or langchain.SequentialChain you can check my article about it in this NLP section here

# Import necessary modules
from langchain.prompts import PromptTemplate
from langchain.chains.sequential import SequentialChain
from langchain.chains.llm import LLMChain
from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler
from langchain.memory import SimpleMemory
import logging

# Initialize logging
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
logging.info("Script started")

# Initialize LangChain and set debug mode
import langchain
langchain.debug = True

# Import LabelStudioCallbackHandler
from langchain_community.callbacks.labelstudio_callback import LabelStudioCallbackHandler

# Initialize the LLM (ChatOpenAI with Ollama)
from langchain_openai import ChatOpenAI

# Initialize the LabelStudioCallbackHandler
handler = LabelStudioCallbackHandler(
    api_key='your_labelstudio_api_key',  # Replace with your actual API key
    url='http://localhost:8080',
    project_name='Test Project',
    mode='chat',
)

# Initialize the LLM
llm = ChatOpenAI(
    api_key="ollama",
    model="llama3.2",
    base_url="http://localhost:11434/v1",
    callbacks=[handler],
)

# Step 1: Process Additional Context and Course Theme
logging.info("Step 1: Setting up course context prompt template")
context_template = """
You are designing a course titled "{course_title}" aimed at {target_audience}.

Additional Information:
{additional_info}

Learning Objectives:
{learning_objectives}

Topics:
{topics}

Provide a concise summary of how the additional information, topics, and learning objectives can be integrated into the course structure.

**Chain of Thought:** Please explain your reasoning step by step before providing the summary.
"""

context_prompt = PromptTemplate(
    input_variables=["course_title", "target_audience", "additional_info", "learning_objectives", "topics"],
    template=context_template
)

context_chain = LLMChain(
    llm=llm,
    prompt=context_prompt,
    output_key="context_summary",
    callbacks=[handler],
)

# Step 2: Generate the Outline
logging.info("Step 2: Setting up outline prompt template")
outline_template = """
Using the topics and context summary below, create a detailed outline for the course titled "{course_title}".

Target Audience:
{target_audience}

Topics:
{topics}

Context Summary:
{context_summary}

Outline Structure:
1. Course Introduction: Brief overview of course goals and structure.
2. Module Introduction: Summarize each module and its learning objectives.
3. Main Content: Present a breakdown of each module, detailing topics covered, key activities, and practical exercises.
4. Assessments: Define types of assessments to evaluate learners’ understanding.
5. Conclusion: Summary of course takeaways and final assignments or assessments.

**Chain of Thought:** Think through the structure carefully and explain your reasoning step by step before providing the outline.
"""

outline_prompt = PromptTemplate(
    input_variables=["course_title", "target_audience", "topics", "context_summary"],
    template=outline_template
)

outline_chain = LLMChain(
    llm=llm,
    prompt=outline_prompt,
    output_key="outline",
    callbacks=[handler],
)

# Step 3: Write the Course Content
logging.info("Step 3: Write the Course Content")
course_content_template = """
Write engaging content for the course titled "{course_title}", aimed at {target_audience}, based on the following outline and topics.

Outline:
{outline}

Topics:
{topics}

Guidelines:
- Ensure content is practical and informative, providing actionable insights on each topic.
- Structure content to suit the needs of {target_audience}, with clear explanations, examples, and exercises.
- Avoid jargon where possible and ensure that all explanations are accessible and engaging.
- Include key takeaways for each module.

**Chain of Thought:** Provide detailed reasoning, explaining your thought process as you develop the course content.
"""

course_content_prompt = PromptTemplate(
    input_variables=["course_title", "target_audience", "outline", "topics"],
    template=course_content_template
)

course_content_chain = LLMChain(
    llm=llm,
    prompt=course_content_prompt,
    output_key="course_content",
    callbacks=[handler],
)

# Step 4: Combine the chains into a SequentialChain to write the course
logging.info("Step 4: Combine the chains into a SequentialChain and generate course content")
memory = SimpleMemory()
overall_chain = SequentialChain(
    chains=[context_chain, outline_chain, course_content_chain],
    input_variables=["course_title", "target_audience", "additional_info", "learning_objectives", "topics"],
    output_variables=["context_summary", "outline", "course_content"],
    memory=memory,
    verbose=True,
    callbacks=[handler],
)

# Prepare input variables
course_info = {
    'title': 'Introduction to Python Programming',
    'audience': 'beginners with no prior programming experience'
}

learning_objectives = """
- Understand the basics of Python programming.
- Develop problem-solving skills through coding.
- Build foundational skills in Python and basic automation.
"""

topics = "variables, data types, control structures, functions, basic I/O, and error handling"

additional_info = "Focus on practical examples, real-world applications, and hands-on exercises to reinforce learning."

# Run the overall chain
result = overall_chain(
    {
        "course_title": course_info['title'],
        "target_audience": course_info['audience'],
        "additional_info": additional_info,
        "learning_objectives": learning_objectives,
        "topics": topics,
    },
    callbacks=[handler],
)

# Print the outputs
print("Context Summary:\n", result['context_summary'])
print("\nOutline:\n", result['outline'])
print("\nCourse Content:\n", result['course_content'])

Wrap it up

In this article we walked through playing with Langchain Ollama LLM and labeling pipeline using Label Studio. From setting up your environment to configuring API integrations.

By connecting the power of an LLM with Label Studio’s intuitive feedback platform, you now have a system that can support detailed assessments of model responses, gather valuable user feedback, and refine your LLM's output over time 🥷

I hope this guide has not only clarified the technical setup but also empowered you with the confidence to extend and customize this pipeline for various applications. Happy LLM building 🤗