chromadb-integration-skills skill

---
name: chromadb-integration-skills
description: Universal ChromaDB integration patterns for semantic search, persistent storage, and pattern matching across all agent types. Use when agents need to store/search large datasets, build knowledge bases, perform semantic analysis, or maintain persistent memory across sessions.
license: MIT
---

# ChromaDB Integration Skills

**Purpose**: This skill teaches agents how to integrate ChromaDB for semantic search, persistent storage, and pattern matching across ANY domain - research, code, trading, legal, documentation, and more.

**Critical Use Case**: When agents need to work with large datasets (1000+ items), perform semantic search, maintain persistent knowledge, or learn from historical patterns, ChromaDB eliminates token limits and enables powerful vector-based retrieval.

**Used By**: All agent types - researchers, developers, traders, legal analysts, documentation writers, QA testers, etc.

---

## When to Use ChromaDB Integration

Use ChromaDB when:
- **Large Datasets**: Working with 1000+ items (documents, code files, bugs, trades, contracts, etc.)
- **Semantic Search**: Finding items by meaning, not just keywords
- **Persistent Memory**: Knowledge needs to survive across sessions, days, months
- **Pattern Matching**: Identifying similar historical cases/patterns for decision-making
- **Cross-Session Learning**: Building institutional knowledge over time
- **Token Limits**: Data too large to fit in context window (100K+ tokens)
- **Aggregation**: Combining results from multiple queries/sources

---

## Core ChromaDB Concepts

### Collections
**Definition**: Named vector databases storing documents with embeddings and metadata

**Naming Strategy**:
- **Domain-based**: `{domain}_{purpose}_{identifier}`
- **Examples**:
  - Research: `research_prior_art_blockchain_2024`, `research_literature_ml_transformers`
  - Code: `codebase_api_endpoints`, `codebase_bug_patterns_auth`
  - Trading: `backtest_results_sma_strategy`, `market_conditions_spy_2024`
  - Legal: `case_law_patent_eligibility`, `contracts_saas_clauses`
  - Documentation: `api_docs_v2`, `architecture_decisions_2024`

### Documents
**Definition**: Text content to be searched semantically

**Best Practices**:
- **Chunk Size**: 200-500 words optimal (too small = context loss, too large = poor granularity)
- **Content Format**: Title + summary + key details (e.g., `"Patent US10123456 - Blockchain Authentication. Abstract: A method for..."`))
- **Deduplication**: Use unique IDs to prevent duplicate storage

### Metadata
**Definition**: Structured data for filtering, not semantic search

**Strategy**:
```javascript
{
  // Temporal filters
  "date": "2024-11-14",
  "year": 2024,
  "month": 11,

  // Categorical filters
  "type": "bug_report",
  "category": "authentication",
  "severity": "high",

  // Numeric filters
  "citations": 42,
  "price": 150.25,
  "performance_score": 0.87,

  // Source tracking
  "source": "github_issue",
  "author": "kim-asplund",
  "url": "https://..."
}
```

### Embeddings
**Definition**: Vector representations enabling semantic similarity

**How It Works**:
- ChromaDB automatically generates embeddings from document text
- Similar meanings → similar vectors → close in vector space
- Distance metrics (cosine, euclidean) measure similarity

---

## Universal ChromaDB Workflow

### Phase 1: Collection Design

```javascript
// Step 1: Design collection strategy based on agent type
const collectionStrategy = {
  research_agent: "One collection per research topic/question",
  code_agent: "Collections by codebase module/feature",
  trading_agent: "Collections by strategy/timeframe/symbol",
  legal_agent: "Collections by practice area/jurisdiction",
  documentation_agent: "Collections by project/version"
};

// Step 2: Create collection with descriptive metadata
mcp__chroma__create_collection({
  collection_name: "{domain}_{purpose}_{identifier}",
  embedding_function_name: "default",  // Uses sentence transformers
  metadata: {
    created_date: "2024-11-14",
    domain: "research|code|trading|legal|docs",
    purpose: "Descriptive purpose",
    total_items: 0,  // Will update
    last_updated: "2024-11-14"
  }
});
```

### Phase 2: Data Ingestion

```javascript
// Step 1: Batch data collection (minimize API calls)
const items = collectAllItems();  // From API, files, database, etc.

// Step 2: Transform to ChromaDB format
const documents = items.map(item => formatDocument(item));
const ids = items.map(item => item.id || generateUniqueId());
const metadatas = items.map(item => extractMetadata(item));

// Step 3: Batch insert (ChromaDB handles chunking automatically)
mcp__chroma__add_documents({
  collection_name: collectionName,
  documents: documents,
  ids: ids,
  metadatas: metadatas
});

// Step 4: Update collection metadata
mcp__chroma__modify_collection({
  collection_name: collectionName,
  new_metadata: {
    ...existingMetadata,
    total_items: items.length,
    last_updated: new Date().toISOString()
  }
});
```

### Phase 3: Semantic Search

```javascript
// Step 1: Formulate semantic query (natural language works!)
const query = "authentication failures in production environment";

// Step 2: Execute semantic search with filters
const results = mcp__chroma__query_documents({
  collection_name: collectionName,
  query_texts: [query],
  n_results: 20,
  where: {
    "$and": [
      { "environment": "production" },
      { "severity": { "$in": ["high", "critical"] } },
      { "date": { "$gte": "2024-01-01" } }
    ]
  },
  include: ["documents", "metadatas", "distances"]
});

// Step 3: Filter by semantic similarity (distance threshold)
const highlyRelevant = results.ids[0].filter((id, idx) =>
  results.distances[0][idx] < 0.3  // Adjust threshold based on use case
);

// Step 4: Retrieve full details if needed
const fullDetails = mcp__chroma__get_documents({
  collection_name: collectionName,
  ids: highlyRelevant,
  include: ["documents", "metadatas"]
});
```

### Phase 4: Pattern Matching

```javascript
// Cross-collection pattern detection
const allCollections = mcp__chroma__list_collections();
const relevantCollections = allCollections.filter(c =>
  c.startsWith(collectionPrefix)
);

const patterns = [];
for (const collection of relevantCollections) {
  const matches = mcp__chroma__query_documents({
    collection_name: collection,
    query_texts: [patternQuery],
    n_results: 10,
    where: { "outcome": "success" }  // Only successful cases
  });

  if (matches.ids[0].length > 0) {
    patterns.push({
      collection: collection,
      matches: matches,
      success_rate: calculateSuccessRate(matches)
    });
  }
}

// Identify best pattern
const bestPattern = patterns.sort((a, b) =>
  b.success_rate - a.success_rate
)[0];
```

---

## Use Case Templates

### Template 1: Research Agent - Literature Review

**Problem**: Store 1000+ research papers, find semantically similar work

```javascript
// Collection: research_literature_{topic}
const papers = fetchPapersFromAPI("machine learning transformers");

mcp__chroma__create_collection({
  collection_name: "research_literature_ml_transformers",
  metadata: { topic: "ML Transformers", papers_count: 0 }
});

// Store papers with rich metadata
papers.forEach(paper => {
  mcp__chroma__add_documents({
    collection_name: "research_literature_ml_transformers",
    documents: [`${paper.title}. ${paper.abstract}`],
    ids: [paper.doi || paper.id],
    metadatas: [{
      title: paper.title,
      authors: paper.authors.join(", "),
      year: paper.year,
      citations: paper.citation_count,
      venue: paper.venue,
      url: paper.url
    }]
  });
});

// Semantic search: "Find papers about attention mechanisms for vision"
const relevant = mcp__chroma__query_documents({
  collection_name: "research_literature_ml_transformers",
  query_texts: ["attention mechanisms computer vision"],
  n_results: 20,
  where: { "year": { "$gte": 2020 }, "citations": { "$gte": 50 } }
});
```

**Benefits**: No token limits, semantic discovery, citation filtering, persistent library

---

### Template 2: Code Agent - Bug Pattern Recognition

**Problem**: Store bug reports, identify similar issues, suggest solutions

```javascript
// Collection: codebase_bug_patterns_{module}
const bugs = fetchAllGitHubIssues("is:issue label:bug");

mcp__chroma__create_collection({
  collection_name: "codebase_bug_patterns_auth",
  metadata: { module: "authentication", total_bugs: 0 }
});

// Store bugs with solutions
bugs.forEach(bug => {
  mcp__chroma__add_documents({
    collection_name: "codebase_bug_patterns_auth",
    documents: [`Bug #${bug.number}: ${bug.title}. ${bug.body}`],
    ids: [`bug_${bug.number}`],
    metadatas: [{
      number: bug.number,
      title: bug.title,
      severity: bug.labels.find(l => l.startsWith("severity:"))?.split(":")[1],
      status: bug.state,
      solution: bug.resolution || "No solution yet",
      created_at: bug.created_at,
      resolved_at: bug.closed_at,
      url: bug.html_url
    }]
  });
});

// New bug arrives - find similar historical bugs
const newBugDescription = "User login fails with 401 error after password reset";

const similarBugs = mcp__chroma__query_documents({
  collection_name: "codebase_bug_patterns_auth",
  query_texts: [newBugDescription],
  n_results: 10,
  where: { "status": "closed", "solution": { "$ne": "No solution yet" } }
});

// Extract solution from most similar resolved bug
const suggestedSolution = similarBugs.metadatas[0][0].solution;
```

**Benefits**: Instant bug pattern matching, solution reuse, similar issue detection

---

### Template 3: Trading Agent - Backtest Results Database

**Problem**: Store 10,000+ backtest results, identify optimal parameter patterns

```javascript
// Collection: backtest_results_{strategy_name}
const backtests = runParameterSweep(strategyCode, parameterRanges);

mcp__chroma__create_collection({
  collection_name: "backtest_results_sma_crossover",
  metadata: { strategy: "SMA Crossover", total_backtests: 0 }
});

// Store each backtest with parameters + results
backtests.forEach(backtest => {
  const description = `
    SMA Crossover strategy with fast=${backtest.params.fast_period},
    slow=${backtest.params.slow_period}, stop_loss=${backtest.params.stop_loss}.
    Market conditions: ${backtest.market_regime}, volatility=${backtest.avg_volatility}.
  `;

  mcp__chroma__add_documents({
    collection_name: "backtest_results_sma_crossover",
    documents: [description],
    ids: [`backtest_${backtest.id}`],
    metadatas: [{
      fast_period: backtest.params.fast_period,
      slow_period: backtest.params.slow_period,
      stop_loss: backtest.params.stop_loss,
      sharpe_ratio: backtest.sharpe_ratio,
      max_drawdown: backtest.max_drawdown,
      win_rate: backtest.win_rate,
      total_return: backtest.total_return,
      market_regime: backtest.market_regime,
      symbol: backtest.symbol,
      timeframe: backtest.timeframe,
      start_date: backtest.start_date,
      end_date: backtest.end_date
    }]
  });
});

// Find optimal parameters for current market conditions
const currentMarket = analyzeCurrentMarket();
const marketDescription = `
  Market regime: ${currentMarket.regime}, volatility: ${currentMarket.volatility},
  trend strength: ${currentMarket.trend_strength}
`;

const optimalBacktests = mcp__chroma__query_documents({
  collection_name: "backtest_results_sma_crossover",
  query_texts: [marketDescription],
  n_results: 20,
  where: {
    "$and": [
      { "sharpe_ratio": { "$gte": 1.5 } },
      { "max_drawdown": { "$lte": -0.15 } },
      { "symbol": currentMarket.symbol }
    ]
  }
});

// Extract best parameter set
const bestParams = optimalBacktests.metadatas[0][0];
```

**Benefits**: Parameter optimization, market regime matching, performance pattern discovery

---

### Template 4: Documentation Agent - Style Guide Enforcement

**Problem**: Store API documentation examples, ensure consistent style

```javascript
// Collection: api_docs_{project_version}
const existingDocs = parseAllApiDocs("./docs/api/");

mcp__chroma__create_collection({
  collection_name: "api_docs_v2",
  metadata: { version: "2.0", total_endpoints: 0 }
});

// Store documentation with style metadata
existingDocs.forEach(doc => {
  mcp__chroma__add_documents({
    collection_name: "api_docs_v2",
    documents: [doc.fullContent],
    ids: [doc.endpoint],
    metadatas: [{
      endpoint: doc.endpoint,
      method: doc.method,
      category: doc.category,
      style_score: doc.styleScore,  // Computed during ingestion
      has_examples: doc.examples.length > 0,
      has_error_codes: doc.errorCodes.length > 0,
      last_updated: doc.lastModified
    }]
  });
});

// New endpoint documented - find similar endpoints for style consistency
const newEndpoint = "POST /api/v2/users/{id}/preferences";

const similarEndpoints = mcp__chroma__query_documents({
  collection_name: "api_docs_v2",
  query_texts: [`${newEndpoint} user preferences update`],
  n_results: 5,
  where: {
    "$and": [
      { "method": "POST" },
      { "style_score": { "$gte": 0.9 } },
      { "has_examples": true }
    ]
  }
});

// Use similar endpoint as template
const template = similarEndpoints.documents[0][0];
```

**Benefits**: Style consistency, template discovery, automated quality checks

---

### Template 5: QA Testing Agent - Test Pattern Library

**Problem**: Store test cases, identify gaps, suggest new tests

```javascript
// Collection: test_cases_{module}
const existingTests = parseTestFiles("./tests/");

mcp__chroma__create_collection({
  collection_name: "test_cases_authentication",
  metadata: { module: "authentication", total_tests: 0 }
});

// Store test cases with coverage metadata
existingTests.forEach(test => {
  mcp__chroma__add_documents({
    collection_name: "test_cases_authentication",
    documents: [`${test.description}. Covers: ${test.coveredScenarios.join(", ")}`],
    ids: [test.id],
    metadatas: [{
      test_type: test.type,  // "unit", "integration", "e2e"
      file_path: test.filePath,
      line_number: test.lineNumber,
      last_run: test.lastRun,
      status: test.lastStatus,
      execution_time_ms: test.executionTime,
      assertions: test.assertionCount
    }]
  });
});

// New feature added - identify missing test coverage
const newFeature = "Password reset with 2FA verification";

const existingCoverage = mcp__chroma__query_documents({
  collection_name: "test_cases_authentication",
  query_texts: [newFeature],
  n_results: 10
});

// If distance > 0.5, probably not covered
const isCovered = existingCoverage.distances[0][0] < 0.5;

if (!isCovered) {
  // Suggest test cases based on similar features
  const similarFeatures = mcp__chroma__query_documents({
    collection_name: "test_cases_authentication",
    query_texts: ["password reset", "2FA verification"],
    n_results: 5
  });

  // Use similar tests as templates
  const testTemplates = similarFeatures.documents[0];
}
```

**Benefits**: Coverage gap detection, test template discovery, pattern-based test generation

---

## Advanced Patterns

### Pattern 1: Multi-Collection Aggregation

**Use Case**: Search across multiple related collections simultaneously

```javascript
// Example: Search all research topics for a cross-cutting concept
const researchCollections = mcp__chroma__list_collections();
const topicCollections = researchCollections.filter(c =>
  c.startsWith("research_literature_")
);

const crossTopicResults = [];
for (const collection of topicCollections) {
  const results = mcp__chroma__query_documents({
    collection_name: collection,
    query_texts: ["transfer learning"],
    n_results: 10
  });

  crossTopicResults.push({
    topic: collection.replace("research_literature_", ""),
    papers: results
  });
}

// Aggregate and rank by relevance across topics
const allPapers = crossTopicResults.flatMap(r =>
  r.papers.ids[0].map((id, idx) => ({
    id: id,
    topic: r.topic,
    distance: r.papers.distances[0][idx],
    metadata: r.papers.metadatas[0][idx]
  }))
);

const rankedPapers = allPapers.sort((a, b) => a.distance - b.distance);
```

### Pattern 2: Hierarchical Collections

**Use Case**: Parent-child relationship between collections

```javascript
// Parent: codebase_architecture_decisions
// Children: codebase_architecture_decisions_{year}

// Create parent collection with aggregated data
mcp__chroma__create_collection({
  collection_name: "codebase_architecture_decisions",
  metadata: { type: "parent", child_collections: [] }
});

// Create child collections by year
[2022, 2023, 2024].forEach(year => {
  mcp__chroma__create_collection({
    collection_name: `codebase_architecture_decisions_${year}`,
    metadata: { type: "child", parent: "codebase_architecture_decisions", year }
  });
});

// Query strategy: Try child first (faster), fallback to parent
const queryYear = 2024;
let results = mcp__chroma__query_documents({
  collection_name: `codebase_architecture_decisions_${queryYear}`,
  query_texts: [query],
  n_results: 10
});

if (results.ids[0].length < 5) {
  // Not enough results in child, query parent
  results = mcp__chroma__query_documents({
    collection_name: "codebase_architecture_decisions",
    query_texts: [query],
    n_results: 10
  });
}
```

### Pattern 3: Temporal Decay

**Use Case**: Prioritize recent items while keeping historical context

```javascript
// Store items with temporal metadata
mcp__chroma__add_documents({
  collection_name: collectionName,
  documents: documents,
  ids: ids,
  metadatas: metadatas.map(m => ({
    ...m,
    timestamp: Date.now(),
    age_days: 0  // Will be updated
  }))
});

// Query with temporal boost
const results = mcp__chroma__query_documents({
  collection_name: collectionName,
  query_texts: [query],
  n_results: 50  // Get more results for re-ranking
});

// Re-rank with temporal decay
const now = Date.now();
const rankedResults = results.ids[0].map((id, idx) => {
  const ageDays = (now - results.metadatas[0][idx].timestamp) / (1000 * 60 * 60 * 24);
  const decayFactor = Math.exp(-ageDays / 30);  // Half-life ~30 days
  const semanticScore = 1 - results.distances[0][idx];
  const combinedScore = semanticScore * 0.7 + decayFactor * 0.3;

  return {
    id: id,
    semantic_score: semanticScore,
    decay_factor: decayFactor,
    combined_score: combinedScore,
    metadata: results.metadatas[0][idx]
  };
}).sort((a, b) => b.combined_score - a.combined_score);
```

---

## Performance Optimization

### Batching Strategy

```javascript
// BAD: One document at a time (slow)
for (const item of items) {
  mcp__chroma__add_documents({
    collection_name: collectionName,
    documents: [item.document],
    ids: [item.id],
    metadatas: [item.metadata]
  });
}

// GOOD: Batch insert (100x faster)
const BATCH_SIZE = 100;
for (let i = 0; i < items.length; i += BATCH_SIZE) {
  const batch = items.slice(i, i + BATCH_SIZE);
  mcp__chroma__add_documents({
    collection_name: collectionName,
    documents: batch.map(item => item.document),
    ids: batch.map(item => item.id),
    metadatas: batch.map(item => item.metadata)
  });
}
```

### Caching Strategy

```javascript
// Check collection exists before creating
const existingCollections = mcp__chroma__list_collections();
if (!existingCollections.includes(collectionName)) {
  mcp__chroma__create_collection({ collection_name: collectionName });
}

// Check document exists before adding
const existing = mcp__chroma__get_documents({
  collection_name: collectionName,
  ids: [documentId]
});

if (!existing.ids || existing.ids.length === 0) {
  // Document doesn't exist, add it
  mcp__chroma__add_documents({ ... });
} else {
  // Document exists, update instead
  mcp__chroma__update_documents({ ... });
}
```

### Query Optimization

```javascript
// Use metadata filters to reduce search space
const results = mcp__chroma__query_documents({
  collection_name: collectionName,
  query_texts: [query],
  n_results: 20,
  where: {
    // Pre-filter with metadata (faster than post-filtering semantic results)
    "date": { "$gte": "2024-01-01" },
    "category": { "$in": ["high_priority", "critical"] }
  }
});

// Only include what you need
const minimalResults = mcp__chroma__query_documents({
  collection_name: collectionName,
  query_texts: [query],
  n_results: 10,
  include: ["metadatas", "distances"]  // Exclude documents if not needed
});
```

---

## Success Criteria

ChromaDB integration is SUCCESSFUL when:

- ✅ **Collections Created**: Meaningful naming, appropriate metadata
- ✅ **Data Ingested**: Batched efficiently, deduplicated
- ✅ **Semantic Search Works**: Returns relevant results (distance < 0.4)
- ✅ **Metadata Filters Applied**: Correctly scopes search space
- ✅ **Performance Optimized**: Batching, caching, minimal queries
- ✅ **Cross-Collection Queries**: When appropriate for use case
- ✅ **Persistent Knowledge**: Data survives across sessions
- ✅ **Pattern Matching**: Identifies similar historical cases
- ✅ **Token Limits Eliminated**: Handles 1000+ items without context overflow

---

**Skill Version**: 1.0
**Created**: 2025-11-14
**Purpose**: Teach universal ChromaDB integration patterns for all agent types
**Target Quality**: 65/70
**Dependencies**: ChromaDB MCP (mcp__chroma__*)
**Universal**: Works for research, code, trading, legal, documentation, QA, and all other domains

---

## 🔴 Error Handling & Resilience (Priority 1)

**Critical for Production**: Prevent data loss, handle failures gracefully

### Retry with Exponential Backoff

```javascript
async function retryWithBackoff(operation, maxRetries = 3) {
  for (let attempt = 0; attempt < maxRetries; attempt++) {
    try {
      return await operation();
    } catch (error) {
      if (attempt === maxRetries - 1) throw error;
      const delay = Math.pow(2, attempt) * 1000;  // 1s, 2s, 4s
      await sleep(delay);
    }
  }
}
```

### Get-or-Create Pattern

```javascript
function getOrCreateCollection(collectionName, metadata = {}) {
  const collections = mcp__chroma__list_collections();

  if (collections.includes(collectionName)) {
    return { created: false, collection_name: collectionName };
  }

  mcp__chroma__create_collection({
    collection_name: collectionName,
    embedding_function_name: "default",
    metadata: metadata
  });

  return { created: true, collection_name: collectionName };
}
```

### Document Validation

```javascript
function validateDocument(document, id, metadata) {
  if (!document || typeof document !== 'string') {
    throw new Error(`Document must be non-empty string`);
  }
  if (!id || id.includes(' ')) {
    throw new Error(`ID must be non-empty string without spaces`);
  }
  if (metadata && typeof metadata !== 'object') {
    throw new Error(`Metadata must be object`);
  }
}
```

**See full patterns**: Load `chromadb-error-handling` sub-skill

---

## 🧪 Testing Patterns (Priority 1)

**Essential for Quality**: Ensure ChromaDB integrations work correctly

### Unit Tests (Mock ChromaDB)

```javascript
// Mock ChromaDB for fast unit tests
class MockChromaDB {
  constructor() {
    this.collections = {};
  }

  create_collection({ collection_name, metadata }) {
    this.collections[collection_name] = {
      documents: [], ids: [], metadatas: [], metadata
    };
  }

  query_documents({ collection_name, query_texts, n_results }) {
    const collection = this.collections[collection_name];
    return {
      ids: [collection.ids.slice(0, n_results)],
      distances: [collection.ids.slice(0, n_results).map(() => 0.2)]
    };
  }
}
```

### Integration Tests (Real ChromaDB)

```javascript
describe('Semantic Search Integration', () => {
  test('returns relevant documents', async () => {
    await chromaClient.add({
      collection_name: testCollection,
      documents: [
        'Machine learning uses neural networks',
        'Python is a programming language'
      ],
      ids: ['doc1', 'doc2']
    });

    const results = await chromaClient.query({
      collection_name: testCollection,
      query_texts: ['neural networks deep learning'],
      n_results: 2
    });

    expect(results.ids[0]).toContain('doc1');
    expect(results.distances[0][0]).toBeLessThan(0.4);
  });
});
```

**See full patterns**: Load `chromadb-testing-patterns` sub-skill

---

## 🔐 Security & Privacy (Priority 2)

**Critical for Compliance**: Protect PII, sanitize data

### PII Redaction

```javascript
function redactPII(text) {
  let redacted = text;

  // Email redaction
  redacted = redacted.replace(
    /\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b/g,
    '[EMAIL_REDACTED]'
  );

  // Phone redaction
  redacted = redacted.replace(/\b\d{3}[-.]?\d{3}[-.]?\d{4}\b/g, '[PHONE_REDACTED]');

  // SSN redaction
  redacted = redacted.replace(/\b\d{3}-\d{2}-\d{4}\b/g, '[SSN_REDACTED]');

  return redacted;
}
```

### Secret Sanitization

```javascript
function redactSecrets(text) {
  let redacted = text;

  // GitHub tokens
  redacted = redacted.replace(/ghp_[a-zA-Z0-9]{36}/g, '[GITHUB_TOKEN]');

  // AWS keys
  redacted = redacted.replace(/AKIA[0-9A-Z]{16}/g, '[AWS_KEY]');

  // API keys
  redacted = redacted.replace(
    /api[_-]?key['\"]?\s*[:=]\s*['\"]?([a-zA-Z0-9_-]{20,})/gi,
    'api_key: [REDACTED]'
  );

  return redacted;
}
```

### Access Control

```javascript
const collectionPermissions = {
  'research_confidential': ['research_team', 'admin'],
  'customer_pii': ['support_team', 'admin']
};

function checkAccess(collectionName, userRole) {
  const allowedRoles = collectionPermissions[collectionName] || ['admin'];

  if (!allowedRoles.includes(userRole)) {
    throw new Error(`Access denied for role '${userRole}'`);
  }
}
```

**See full patterns**: Load `chromadb-security-patterns` sub-skill

---

## 🗄️ Data Lifecycle Management (Priority 2)

**Sustain Production**: Version collections, archive old data

### Schema Versioning

```javascript
// Semantic versioning for collections
function createVersionedCollection(domain, purpose, version = 'v1') {
  const collectionName = `${domain}_${purpose}_${version}`;

  mcp__chroma__create_collection({
    collection_name: collectionName,
    metadata: {
      version: version,
      created_at: new Date().toISOString(),
      schema_version: '1.0',
      retention_days: 730,  // 2 years
      lifecycle_stage: 'active'
    }
  });

  return collectionName;
}
```

### Schema Migration

```javascript
async function migrateCollectionSchema(oldCollection, newVersion) {
  const newCollection = `${oldCollection}_v${newVersion}`;

  // Create new collection
  await mcp__chroma__create_collection({
    collection_name: newCollection,
    metadata: { migrated_from: oldCollection, version: newVersion }
  });

  // Copy all documents with transformed metadata
  const allDocs = await mcp__chroma__get_documents({
    collection_name: oldCollection,
    limit: 100000
  });

  const transformedMetadatas = allDocs.metadatas.map(transformMetadata);

  await mcp__chroma__add_documents({
    collection_name: newCollection,
    documents: allDocs.documents,
    ids: allDocs.ids,
    metadatas: transformedMetadatas
  });

  // Mark old as deprecated
  await mcp__chroma__modify_collection({
    collection_name: oldCollection,
    new_metadata: { lifecycle_stage: 'deprecated', replacement: newCollection }
  });
}
```

### Retention Enforcement

```javascript
async function enforceRetentionPolicies() {
  const collections = await mcp__chroma__list_collections();

  for (const collectionName of collections) {
    const info = await mcp__chroma__get_collection_info({ collection_name });
    const retentionDays = info.metadata.retention_days || 730;
    const ageDays = calculateAgeDays(info.metadata.created_at);

    if (ageDays > retentionDays) {
      await archiveCollection(collectionName);  // Backup first
      await mcp__chroma__delete_collection({ collection_name: collectionName });
    }
  }
}
```

**See full patterns**: Load `chromadb-lifecycle-management` sub-skill

---

## 🐛 Debugging & Troubleshooting (Priority 1)

### Common Issues

**Issue: No results returned**
- **Cause**: Distance threshold too strict, wrong collection
- **Fix**: Increase threshold (0.3 → 0.5), verify collection name
- **Debug**: Check `results.distances[0]` values

**Issue: Poor semantic matches**
- **Cause**: Document chunking too large/small
- **Fix**: Optimal chunk size 200-500 words
- **Debug**: Review document length, split long documents

**Issue: Slow queries**
- **Cause**: Large collection without metadata filters
- **Fix**: Add metadata pre-filters (`where` clause)
- **Debug**: Check collection size, add filters

### Distance Threshold Guide

| Distance | Similarity | Use Case |
|----------|-----------|----------|
| < 0.2 | Almost exact | Duplicate detection |
| 0.2-0.3 | Very similar | High precision search |
| 0.3-0.5 | Moderately similar | Balanced search |
| 0.5-0.7 | Weakly similar | Broad exploration |
| > 0.7 | Different topics | Not relevant |

### Antipatterns to Avoid

❌ **Storing entire files as single document**
- Loses granularity, poor search relevance
- ✅ **Fix**: Chunk into 200-500 word sections

❌ **No metadata filters on large collections**
- Slow queries, high latency
- ✅ **Fix**: Always filter by date, category, type

❌ **Not deduplicating documents**
- Wasted storage, duplicate results
- ✅ **Fix**: Check existence before adding

❌ **Ignoring connection failures**
- Data loss, silent failures
- ✅ **Fix**: Implement retry logic, fallback

---

## 📚 Sub-Skill Reference

Load targeted sub-skills for deep dives:

- **chromadb-error-handling**: Retry patterns, validation, circuit breakers (~150 lines)
- **chromadb-testing-patterns**: Unit/integration tests, mocking, fixtures (~120 lines)
- **chromadb-security-patterns**: PII redaction, access control, GDPR compliance (~90 lines)
- **chromadb-lifecycle-management**: Versioning, migration, archival, retention (~100 lines)

**Usage**: `Skill({ skill: "chromadb-error-handling" })`

---

## Updated Success Criteria

ChromaDB integration is **PRODUCTION-READY** when:

**Core Functionality** (Original):
- ✅ Collections created with meaningful naming
- ✅ Data ingested efficiently (batching)
- ✅ Semantic search returns relevant results
- ✅ Metadata filters applied correctly
- ✅ Performance optimized

**Production Readiness** (New):
- ✅ **Error Handling**: Retry logic, validation, graceful degradation
- ✅ **Testing**: Unit tests (mocked), integration tests (real ChromaDB)
- ✅ **Security**: PII redacted, secrets sanitized, access control
- ✅ **Lifecycle**: Versioning strategy, retention policies, archival

**Quality Score**: 65/70 → **85/100** (with all enhancements)

---

**Skill Version**: 2.0
**Updated**: 2025-11-14
**Enhancements**: Error handling, testing, security, lifecycle management
**Quality Score**: 85/100 (Production-Ready)
**Dependencies**: ChromaDB MCP (mcp__chroma__*)
**Sub-Skills**: 4 modular sub-skills for targeted loading