playbooks

home / skills / basedhardware / omi / omi-backend-patterns

omi-backend-patterns skill

/.cursor/skills/omi-backend-patterns

This skill guides backend development for Omi conversations, memory extraction, and LangGraph integration across Firestore, Pinecone, and Redis.

npx playbooks add skill basedhardware/omi --skill omi-backend-patterns

Review the files below or copy the command above to add this skill to your agents.

Files (7)
SKILL.md
5.1 KB
---
name: omi-backend-patterns
description: "Omi backend patterns conversation processing memory extraction chat system LangGraph FastAPI Firestore Pinecone Redis"
---

# Omi Backend Patterns Skill

This skill provides guidance for working with the Omi backend, including conversation processing, memory extraction, chat system, and LangGraph integration.

## When to Use

Use this skill when:
- Working on backend Python code in `backend/`
- Implementing new API endpoints
- Processing conversations or extracting memories
- Working with the LangGraph chat system
- Integrating with Firestore, Pinecone, or Redis

## Key Patterns

### Conversation Processing

The conversation processing pipeline follows this flow:

1. **Audio arrives via WebSocket** (`/v4/listen`)
2. **Transcription** via Deepgram/Soniox/Speechmatics
3. **Conversation creation** in Firestore (status: "in_progress")
4. **Processing trigger** via `POST /v1/conversations` or timeout
5. **LLM extraction** of structured data:
   - Title and overview
   - Action items
   - Calendar events
   - Memories (user facts)
6. **Storage** in Firestore and Pinecone

**Key Function**: `utils/conversations/process_conversation.py::process_conversation()`

### Memory Extraction

Memories are extracted from conversations using LLM:

```python
from utils.llm.conversation_processing import _extract_memories

memories = await _extract_memories(
    transcript=transcript,
    existing_memories=existing_memories,
)
```

**Categories**: personal, health, work, relationships, preferences

### Chat System Architecture

The chat system uses LangGraph for routing:

1. **Classification**: `requires_context()` determines path
2. **Simple Path**: Direct LLM response (no context needed)
3. **Agentic Path**: Full tool access with LangGraph ReAct agent
4. **Persona Path**: Persona app responses

**Key File**: `utils/retrieval/graph.py`

### Module Hierarchy

**CRITICAL**: Always follow the import hierarchy:

1. `database/` - Data access (lowest)
2. `utils/` - Business logic
3. `routers/` - API endpoints
4. `main.py` - Application entry

**Never import from higher levels in lower levels!**

### Database Patterns

- **Firestore**: Primary database for conversations, memories, users
- **Pinecone**: Vector embeddings for semantic search
- **Redis**: Caching (speech profiles, enabled apps, user names)
- **GCS**: Binary files (audio, photos, speech profiles)

### API Endpoint Patterns

- Use FastAPI routers in `routers/`
- Keep routers thin - business logic in `utils/`
- Use dependency injection for auth
- Return consistent error formats

## Common Tasks

### Adding a New API Endpoint

1. Create router function in appropriate `routers/*.py`
2. Add business logic in `utils/`
3. Use database functions from `database/`
4. Follow error handling patterns
5. Add to router in `main.py`

### Processing Conversations

1. Use `process_conversation()` from `utils/conversations/process_conversation.py`
2. Handle extraction results
3. Store in Firestore and Pinecone
4. Trigger app webhooks if needed

### Adding a Chat Tool

1. Create tool function in `utils/retrieval/tools/`
2. Use `@tool` decorator from LangChain
3. Add to tool loading in `utils/retrieval/tools/app_tools.py`
4. Tool will be available in agentic chat path

## Related Documentation

**The `docs/` folder is the single source of truth for all user-facing documentation, deployed at [docs.omi.me](https://docs.omi.me/).**

- **Backend Deep Dive**: `docs/doc/developer/backend/backend_deepdive.mdx` - [View online](https://docs.omi.me/doc/developer/backend/backend_deepdive)
- **Chat System**: `docs/doc/developer/backend/chat_system.mdx` - [View online](https://docs.omi.me/doc/developer/backend/chat_system)
- **Data Storage**: `docs/doc/developer/backend/StoringConversations.mdx` - [View online](https://docs.omi.me/doc/developer/backend/StoringConversations)
- **Transcription**: `docs/doc/developer/backend/transcription.mdx` - [View online](https://docs.omi.me/doc/developer/backend/transcription)
- **Backend Setup**: `docs/doc/developer/backend/Backend_Setup.mdx` - [View online](https://docs.omi.me/doc/developer/backend/Backend_Setup)
- **Backend Architecture**: `.cursor/rules/backend-architecture.mdc`

## Related Cursor Resources

### Rules
- `.cursor/rules/backend-architecture.mdc` - System architecture and module hierarchy
- `.cursor/rules/backend-api-patterns.mdc` - FastAPI router patterns
- `.cursor/rules/backend-database-patterns.mdc` - Database storage patterns
- `.cursor/rules/backend-llm-patterns.mdc` - LLM integration patterns
- `.cursor/rules/backend-testing.mdc` - Testing patterns
- `.cursor/rules/backend-imports.mdc` - Import rules
- `.cursor/rules/memory-management.mdc` - Memory management

### Subagents
- `.cursor/agents/backend-api-developer/` - Uses this skill for API development
- `.cursor/agents/backend-llm-engineer/` - Uses this skill for LLM integration
- `.cursor/agents/backend-database-engineer/` - Uses this skill for database work

### Commands
- `/backend-setup` - Uses this skill for setup guidance
- `/backend-test` - Uses this skill for testing patterns
- `/backend-deploy` - Uses this skill for deployment patterns

Overview

This skill documents backend patterns for the Omi system, focused on conversation processing, memory extraction, chat routing, and integrations with LangGraph, Firestore, Pinecone, and Redis. It summarizes module hierarchy, API and database conventions, and the typical conversation-to-memory pipeline. Use it as a practical guide when implementing or extending backend features in Python/FastAPI services.

How this skill works

The skill explains the end-to-end conversation pipeline: audio arrives via WebSocket, transcription runs through supported providers, the system creates a conversation record, and processing is triggered by an API call or timeout. LLMs extract structured artifacts (title, action items, events, and categorized memories), which are stored in Firestore and indexed in Pinecone; Redis and GCS are used for caching and binary storage. Chat routing uses LangGraph to choose between direct LLM responses, agentic tool-enabled paths, or persona-driven responses.

When to use it

  • Building or modifying backend endpoints under routers/
  • Implementing conversation processing or transcription flows
  • Adding memory extraction or LLM-based extractors
  • Integrating vector search with Pinecone or semantic storage in Firestore
  • Adding or extending LangGraph chat routing or tools

Best practices

  • Follow strict import hierarchy: database/ → utils/ → routers/ → main.py; avoid upward imports.
  • Keep routers thin; place business logic in utils/ and DB access in database/.
  • Use dependency injection for auth and consistent error response shapes.
  • Use process_conversation() as the canonical pipeline entry for conversation processing.
  • Store structured LLM outputs in Firestore and index embeddings in Pinecone; use Redis for ephemeral caches.

Example use cases

  • Add a new FastAPI endpoint that triggers conversation processing and returns a job ID.
  • Enhance memory extraction to capture new categories such as medications or travel preferences.
  • Register a new LangGraph tool and expose it through the agentic chat path for external data access.
  • Implement webhook triggers after conversation processing to notify client apps or wearable devices.
  • Migrate a binary storage flow to GCS for audio and profile assets while keeping metadata in Firestore.

FAQ

Where is the main conversation processor located?

The canonical function is utils/conversations/process_conversation.py::process_conversation().

How do I add a new chat tool for agentic flows?

Create the tool in utils/retrieval/tools/, decorate with @tool, and register it in utils/retrieval/tools/app_tools.py so LangGraph can load it.