sf-ai-agentforce-observability skill

---
name: sf-ai-agentforce-observability
description: >
  Extract and analyze Agentforce session tracing data from Salesforce Data 360.
  Supports high-volume extraction (1-10M records/day), Polars-based analysis,
  and debugging workflows for agent sessions.
license: MIT
compatibility: "Requires Data 360 enabled org with Agentforce Session Tracing"
metadata:
  version: "1.0.0"
  author: "Jag Valaiyapathy"
  data_model: "Session Tracing Data Model (STDM)"
  storage_format: "Parquet (via PyArrow)"
  analysis_library: "Polars"
hooks:
  PreToolUse:
    - matcher: Bash
      hooks:
        - type: command
          command: "python3 ${SHARED_HOOKS}/scripts/guardrails.py"
          timeout: 5000
  PostToolUse:
    - matcher: "Write|Edit"
      hooks:
        - type: command
          command: "python3 ${SKILL_HOOKS}/validate-extraction.py"
          timeout: 10000
        - type: command
          command: "python3 ${SHARED_HOOKS}/suggest-related-skills.py sf-ai-agentforce-observability"
          timeout: 5000
  SubagentStop:
    - type: command
      command: "python3 ${SHARED_HOOKS}/scripts/chain-validator.py sf-ai-agentforce-observability"
      timeout: 5000
---

<!-- TIER: 1 | ENTRY POINT -->
<!-- This is the starting document - read this FIRST -->
<!-- Pattern: Follows sf-data for Python extraction scripts -->

# sf-ai-agentforce-observability: Agentforce Session Tracing Extraction & Analysis

Expert in extracting and analyzing Agentforce session tracing data from Salesforce Data 360. Supports high-volume data extraction (1-10M records/day), Parquet storage, and Polars-based analysis for debugging agent behavior.

## Core Responsibilities

1. **Session Extraction**: Extract STDM (Session Tracing Data Model) data via Data 360 Query API
2. **Data Storage**: Write to Parquet format with PyArrow for efficient storage
3. **Analysis**: Polars-based lazy evaluation for memory-efficient analysis
4. **Debugging**: Session timeline reconstruction for troubleshooting agent issues
5. **Cross-Skill Integration**: Works with sf-connected-apps for auth, sf-ai-agentscript for fixes

## Document Map

| Need | Document | Description |
|------|----------|-------------|
| **Quick start** | [README.md](README.md) | Installation & basic usage |
| **Data model** | [resources/data-model-reference.md](resources/data-model-reference.md) | Full STDM schema documentation |
| **Query patterns** | [resources/query-patterns.md](resources/query-patterns.md) | Data Cloud SQL examples |
| **Analysis recipes** | [resources/analysis-cookbook.md](resources/analysis-cookbook.md) | Common Polars patterns |
| **CLI reference** | [docs/cli-reference.md](docs/cli-reference.md) | Complete command documentation |
| **Auth setup** | [docs/auth-setup.md](docs/auth-setup.md) | JWT Bearer configuration |
| **Troubleshooting** | [resources/troubleshooting.md](resources/troubleshooting.md) | Common issues & fixes |

**Quick Links:**
- [Data Model Overview](#session-tracing-data-model-stdm)
- [CLI Quick Reference](#cli-quick-reference)
- [Analysis Examples](#analysis-examples)
- [Cross-Skill Integration](#cross-skill-integration)

---

## CRITICAL: Prerequisites Checklist

Before extracting session data, verify:

| Check | How to Verify | Why |
|-------|---------------|-----|
| **Data 360 enabled** | Setup → Data 360 | Required for Query API |
| **Salesforce Standard Data Model v1.124+** | Setup → Apps → Packaging → Installed Packages | Required for session tracing DMOs |
| **Einstein Generative AI enabled** | Setup → Einstein Generative AI | Enables agent capabilities |
| **Session Tracing enabled** | Setup → Einstein Audit, Analytics, and Monitoring | Must toggle ON to collect data |
| **JWT Auth configured** | Use `sf-connected-apps` | Required for Data 360 API |

> **Official Setup Guide**: [Set Up Agentforce Session Tracing](https://help.salesforce.com/s/articleView?id=ai.generative_ai_session_trace_setup.htm)

### Auth Setup (via sf-connected-apps)

```bash
# 1. Create key directory
mkdir -p ~/.sf/jwt

# 2. Generate certificate (naming convention: {org}-agentforce-observability)
openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 \
  -keyout ~/.sf/jwt/myorg-agentforce-observability.key \
  -out ~/.sf/jwt/myorg-agentforce-observability.crt \
  -subj "/CN=AgentforceObservability/O=MyOrg"

# 3. Secure the private key
chmod 600 ~/.sf/jwt/myorg-agentforce-observability.key

# 4. Create External Client App in Salesforce (see docs/auth-setup.md)
# Required scopes: cdp_query_api, refresh_token/offline_access
```

**Key Path Resolution Order:**
1. Explicit `--key-path` argument
2. App-specific: `~/.sf/jwt/{org}-agentforce-observability.key`
3. Generic fallback: `~/.sf/jwt/{org}.key`

See [docs/auth-setup.md](docs/auth-setup.md) for detailed instructions.

---

## T6 Live API Discovery Summary ✅

**Validated: January 30, 2026** | **24 DMOs Found** | **260+ Test Points**

| Category | DMOs | Status |
|----------|------|--------|
| **Session Tracing** | 5 | ✅ All Found (Session, Interaction, Step, Message, Participant) |
| **Agent Optimizer** | 6 | ✅ All Found (Moment, Tag system) |
| **GenAI Audit** | 13 | ✅ All Found (Generation, Quality, Feedback, Gateway) |
| **RAG Quality** | 3 | ❌ Not Found (GenAIRetriever* DMOs don't exist) |

**Key Discoveries:**
- Field naming: API uses `AiAgent` (lowercase 'i'), not `AIAgent`
- Agent name location: Stored on `Moment`, not `Session`
- Channel types: `E & O`, `Builder`, `SCRT2 - EmbeddedMessaging`, `Voice`, `NGC`, `Builder: Voice Preview`
- Agent types: `EinsteinServiceAgent`, `AgentforceEmployeeAgent`, `AgentforceServiceAgent`, `Employee`
- Participant roles: `USER`, `AGENT` (not Owner/Observer)
- GenAI detectors: `TOXICITY` (9 categories), `PII` (4 types), `PROMPT_DEFENSE`, `InstructionAdherence`

---

## Session Tracing Data Model (STDM)

The STDM consists of 5 core DMOs plus 13 GenAI Audit DMOs. **Important**: Field names use `AiAgent` (lowercase 'i'), not `AIAgent`.

```
ssot__AIAgentSession__dlm (SESSION)
├── ssot__Id__c                          # Session ID (UUID)
├── ssot__StartTimestamp__c              # Session start (TimestampTZ)
├── ssot__EndTimestamp__c                # Session end (TimestampTZ)
├── ssot__AiAgentSessionEndType__c       # End type (Completed, Abandoned, etc.)
├── ssot__AiAgentChannelType__c          # Channel (PSTN, Messaging, etc.)
├── ssot__RelatedMessagingSessionId__c   # Linked messaging session
├── ssot__RelatedVoiceCallId__c          # Linked voice call
├── ssot__SessionOwnerId__c              # Owner ID
├── ssot__IndividualId__c                # Data Cloud individual
└── ssot__InternalOrganizationId__c      # Org ID

    └── ssot__AIAgentInteraction__dlm (TURN)  [1:N]
        ├── ssot__Id__c                          # Interaction ID
        ├── ssot__AiAgentSessionId__c            # FK to Session
        ├── ssot__AiAgentInteractionType__c      # TURN or SESSION_END
        ├── ssot__TopicApiName__c                # Topic that handled this turn
        ├── ssot__StartTimestamp__c              # Turn start
        ├── ssot__EndTimestamp__c                # Turn end
        ├── ssot__TelemetryTraceId__c            # Trace ID for debugging
        └── ssot__TelemetryTraceSpanId__c        # Span ID for debugging

            └── ssot__AIAgentInteractionStep__dlm (STEP)  [1:N]
                ├── ssot__Id__c                          # Step ID
                ├── ssot__AiAgentInteractionId__c        # FK to Interaction
                ├── ssot__AiAgentInteractionStepType__c  # LLM_STEP or ACTION_STEP
                ├── ssot__Name__c                        # Action/step name
                ├── ssot__InputValueText__c              # Input to step (JSON)
                ├── ssot__OutputValueText__c             # Output from step (JSON)
                ├── ssot__ErrorMessageText__c            # Error if step failed
                ├── ssot__PreStepVariableText__c         # Variables before
                ├── ssot__PostStepVariableText__c        # Variables after
                ├── ssot__GenerationId__c                # LLM generation ID
                └── ssot__GenAiGatewayRequestId__c       # GenAI Gateway request

ssot__AIAgentMoment__dlm (MOMENT - links to Session, not Interaction)
├── ssot__Id__c                          # Moment ID
├── ssot__AiAgentSessionId__c            # FK to Session (NOT interaction!)
├── ssot__AiAgentApiName__c              # Agent API name (lives here!)
├── ssot__AiAgentVersionApiName__c       # Agent version
├── ssot__RequestSummaryText__c          # User request summary
├── ssot__ResponseSummaryText__c         # Agent response summary
├── ssot__StartTimestamp__c              # Moment start
└── ssot__EndTimestamp__c                # Moment end
```

**Key Schema Notes:**
- Agent API name is in `AIAgentMoment`, not `AIAgentSession`
- Moments link to sessions via `AiAgentSessionId`, not interactions
- All field names use `AiAgent` prefix (lowercase 'i')

### GenAI Trust Layer DMOs (13) ✅ T6 Verified

```
GenAIGatewayRequest__dlm (30 fields) - LLM request details
├── gatewayRequestId__c, prompt__c, maskedPrompt__c
├── model__c, provider__c, temperature__c
├── promptTokens__c, completionTokens__c, totalTokens__c
├── enableInputSafetyScoring__c, enableOutputSafetyScoring__c, enablePiiMasking__c
└── sessionId__c, userId__c, appType__c, feature__c

GenAIGeneration__dlm (11 fields) - LLM output
├── generationId__c (FK for Steps)
├── responseText__c, maskedResponseText__c
└── Links to: AIAgentInteractionStep.ssot__GenerationId__c

GenAIContentQuality__dlm (10 fields) - Trust Layer assessment
└── isToxicityDetected__c, parent__c (FK to Generation)

GenAIContentCategory__dlm (10 fields) - Detector results
├── detectorType__c: TOXICITY | PII | PROMPT_DEFENSE | InstructionAdherence
├── category__c: hate, identity, CREDIT_CARD, EMAIL_ADDRESS, High, Low, etc.
└── value__c: Confidence score (0.0-1.0)

GenAIFeedback__dlm (16 fields) - User feedback
├── feedback__c: GOOD | BAD
└── GenAIFeedbackDetail__dlm (10 fields) - Free-text comments
```

**Detector Categories (Live API Verified):**
| Detector | Categories |
|----------|------------|
| `TOXICITY` | `hate`, `identity`, `physical`, `profanity`, `safety_score`, `sexual`, `toxicity`, `violence` |
| `PII` | `CREDIT_CARD`, `EMAIL_ADDRESS`, `PERSON`, `US_PHONE_NUMBER` |
| `PROMPT_DEFENSE` | `aggregatePromptAttackScore`, `isPromptAttackDetected` |
| `InstructionAdherence` | `High`, `Low`, `Uncertain` |

See [resources/data-model-reference.md](resources/data-model-reference.md) for full field documentation.

---

## Workflow (5-Phase Pattern)

### Phase 1: Requirements Gathering

Use **AskUserQuestion** to gather:

| # | Question | Options |
|---|----------|---------|
| 1 | Target org | Org alias from `sf org list` |
| 2 | Time range | Last N days / Date range |
| 3 | Agent filter | All agents / Specific API names |
| 4 | Output format | Parquet (default) / CSV |
| 5 | Analysis type | Summary / Debug session / Full extraction |

### Phase 2: Auth Configuration

Verify JWT auth is configured:

```python
from scripts.auth import Data360Auth

auth = Data360Auth(
    org_alias="myorg",
    consumer_key="YOUR_CONSUMER_KEY"
)

# Test authentication
token = auth.get_token()
print(f"Auth successful: {token[:20]}...")
```

If auth fails, invoke:
```
Skill(skill="sf-connected-apps", args="Setup JWT Bearer for Data 360")
```

### Phase 3: Extraction

**Basic Extraction (last 7 days):**
```bash
python3 scripts/cli.py extract \
  --org prod \
  --days 7 \
  --output ./stdm_data
```

**Filtered Extraction:**
```bash
python3 scripts/cli.py extract \
  --org prod \
  --since 2026-01-01 \
  --until 2026-01-28 \
  --agent Customer_Support_Agent \
  --output ./stdm_data
```

**Session Tree (specific session):**
```bash
python3 scripts/cli.py extract-tree \
  --org prod \
  --session-id "a0x..." \
  --output ./debug_session
```

### Phase 4: Analysis

**Session Summary:**
```python
from scripts.analyzer import STDMAnalyzer
from pathlib import Path

analyzer = STDMAnalyzer(Path("./stdm_data"))

# High-level summary
summary = analyzer.session_summary()
print(summary)

# Step distribution by agent
steps = analyzer.step_distribution(agent_name="Customer_Support_Agent")
print(steps)

# Topic routing analysis
topics = analyzer.topic_analysis()
print(topics)
```

**Debug Specific Session:**
```bash
python3 scripts/cli.py debug-session \
  --data-dir ./stdm_data \
  --session-id "a0x..."
```

### Phase 5: Integration & Next Steps

Based on analysis findings:

| Finding | Next Step | Skill |
|---------|-----------|-------|
| Topic mismatch | Improve topic descriptions | `sf-ai-agentscript` |
| Action failures | Debug Flow/Apex | `sf-flow`, `sf-debug` |
| Slow responses | Optimize actions | `sf-apex` |
| Missing coverage | Add test cases | `sf-ai-agentforce-testing` |

---

## CLI Quick Reference

### Extraction Commands

| Command | Purpose | Example |
|---------|---------|---------|
| `extract` | Extract session data | `extract --org prod --days 7` |
| `extract-tree` | Extract full session tree | `extract-tree --org prod --session-id "a0x..."` |
| `extract-incremental` | Resume from last run | `extract-incremental --org prod` |

### Analysis Commands

| Command | Purpose | Example |
|---------|---------|---------|
| `analyze` | Generate summary stats | `analyze --data-dir ./stdm_data` |
| `debug-session` | Timeline view | `debug-session --session-id "a0x..."` |
| `topics` | Topic analysis | `topics --data-dir ./stdm_data` |

### Common Flags

| Flag | Description | Default |
|------|-------------|---------|
| `--org` | Target org alias | Required |
| `--consumer-key` | ECA consumer key | `$SF_CONSUMER_KEY` env var |
| `--key-path` | JWT private key path | `~/.sf/jwt/{org}-agentforce-observability.key` |
| `--days` | Last N days | 7 |
| `--since` | Start date (YYYY-MM-DD) | - |
| `--until` | End date (YYYY-MM-DD) | Today |
| `--agent` | Filter by agent API name | All |
| `--output` | Output directory | `./stdm_data` |
| `--verbose` | Detailed logging | False |
| `--format` | Output format (table/json/csv) | table |

See [docs/cli-reference.md](docs/cli-reference.md) for complete documentation.

---

## Analysis Examples

### Session Summary

```
📊 SESSION SUMMARY
════════════════════════════════════════════════════════════════

Period: 2026-01-21 to 2026-01-28
Total Sessions: 15,234
Unique Agents: 3

SESSIONS BY AGENT
────────────────────────────────────────────────────────────────
Agent                          │ Sessions │ Avg Turns │ Avg Duration
───────────────────────────────┼──────────┼───────────┼─────────────
Customer_Support_Agent         │   8,502  │    4.2    │     3m 15s
Order_Tracking_Agent           │   4,128  │    2.8    │     1m 45s
Product_FAQ_Agent              │   2,604  │    1.9    │       45s

END TYPE DISTRIBUTION
────────────────────────────────────────────────────────────────
✅ Completed:    12,890 (84.6%)
🔄 Escalated:     1,523 (10.0%)
❌ Abandoned:       821 (5.4%)
```

### Debug Session Timeline

```
🔍 SESSION DEBUG: a0x1234567890ABC
════════════════════════════════════════════════════════════════

Agent: Customer_Support_Agent
Started: 2026-01-28 10:15:23 UTC
Duration: 4m 32s
End Type: Completed
Turns: 5

TIMELINE
────────────────────────────────────────────────────────────────
10:15:23 │ [INPUT]  "I need help with my order #12345"
10:15:24 │ [TOPIC]  → Order_Tracking (confidence: 0.95)
10:15:24 │ [STEP]   LLM_STEP: Identify intent
10:15:25 │ [STEP]   ACTION_STEP: Get_Order_Status
         │          Input: {"orderId": "12345"}
         │          Output: {"status": "Shipped", "eta": "2026-01-30"}
10:15:26 │ [OUTPUT] "Your order #12345 has shipped and will arrive by Jan 30."

10:16:01 │ [INPUT]  "Can I change the delivery address?"
10:16:02 │ [TOPIC]  → Order_Tracking (same topic)
10:16:02 │ [STEP]   LLM_STEP: Clarify request
10:16:03 │ [STEP]   ACTION_STEP: Check_Modification_Eligibility
         │          Input: {"orderId": "12345", "type": "address_change"}
         │          Output: {"eligible": false, "reason": "Already shipped"}
10:16:04 │ [OUTPUT] "I'm sorry, the order has already shipped..."
```

---

## Cross-Skill Integration

### Prerequisite Skills

| Skill | When | How to Invoke |
|-------|------|---------------|
| `sf-connected-apps` | Auth setup | `Skill(skill="sf-connected-apps", args="JWT Bearer for Data Cloud")` |

### Follow-up Skills

| Finding | Skill | How to Invoke |
|---------|-------|---------------|
| Topic routing issues | `sf-ai-agentscript` | `Skill(skill="sf-ai-agentscript", args="Fix topic: [issue]")` |
| Action failures | `sf-flow` / `sf-debug` | `Skill(skill="sf-debug", args="Analyze agent action failure")` |
| Test coverage gaps | `sf-ai-agentforce-testing` | `Skill(skill="sf-ai-agentforce-testing", args="Add test cases")` |

### Commonly Used With

| Skill | Use Case | Confidence |
|-------|----------|------------|
| `sf-ai-agentscript` | Fix agent based on trace analysis | ⭐⭐⭐ Required |
| `sf-ai-agentforce-testing` | Create test cases from observed patterns | ⭐⭐ Recommended |
| `sf-debug` | Deep-dive into action failures | ⭐⭐ Recommended |

---

## Key Insights

| Insight | Description | Action |
|---------|-------------|--------|
| **STDM is read-only** | Data 360 stores traces; cannot modify | Use for analysis only |
| **Session lag** | Data may lag 5-15 minutes | Don't expect real-time |
| **Volume limits** | Query API: 10M records/day | Use incremental extraction |
| **Parquet efficiency** | 10x smaller than JSON | Always use Parquet for storage |
| **Lazy evaluation** | Polars scans without loading | Handles 100M+ rows |
| **~24 records per LLM call** | Each round-trip generates ~24 records | Factor into volume estimates |
| **5-minute collection interval** | Data collection runs every 5 minutes | Account for processing delay |

---

## Billing Considerations

> **Reference**: [Billing Considerations for Agentforce Session Tracing](https://help.salesforce.com/s/articleView?id=ai.generative_ai_session_trace_usage_types.htm)

Agentforce Session Tracing consumes **Data 360 credits** for ingestion, storage, and processing.

### Credit Consumption

| Usage Type | Digital Wallet Card | Description |
|------------|---------------------|-------------|
| **Batch Data Pipeline** | Data Services | Records ingested via data streams. ~24 records per LLM round-trip. **Primary cost driver**. |
| **Data Queries** | Data Services | Records processed when running queries, reports, dashboards |
| **Streaming Calculated Insights** | Data Services | Used for Prompt Builder usage and feedback metrics |
| **Storage Beyond Allocation** | Data Storage | Storage consumed above allocated amount |

### Cost Estimation

```
Records per session ≈ Turns × 24 (avg per LLM call)
Daily records ≈ Sessions/day × Avg turns × 24

Example:
  1,000 sessions/day × 4 turns × 24 = 96,000 records/day ingested
```

**Tip**: Use [Digital Wallet](https://help.salesforce.com/s/articleView?id=sf.digital_wallet.htm) for near real-time consumption tracking.

---

## Common Issues & Fixes

| Error | Cause | Fix |
|-------|-------|-----|
| `401 Unauthorized` | JWT auth expired/invalid | Refresh token or reconfigure ECA |
| `No session data` | Tracing not enabled | Enable Session Tracing in Agent Settings |
| `Query timeout` | Too much data | Add date filters, use incremental |
| `Memory error` | Loading all data | Use Polars lazy frames |
| `Missing DMO` | Wrong API version | Use API v60.0+ |

See [resources/troubleshooting.md](resources/troubleshooting.md) for detailed solutions.

---

## Output Directory Structure

After extraction:

```
stdm_data/
├── sessions/
│   └── date=2026-01-28/
│       └── part-0000.parquet
├── interactions/
│   └── date=2026-01-28/
│       └── part-0000.parquet
├── steps/
│   └── date=2026-01-28/
│       └── part-0000.parquet
├── messages/
│   └── date=2026-01-28/
│       └── part-0000.parquet
└── metadata/
    ├── extraction.json      # Extraction parameters
    └── watermark.json       # For incremental extraction
```

---

## Dependencies

**Python 3.10+** with:

```
polars>=1.0.0           # DataFrame library (lazy evaluation)
pyarrow>=15.0.0         # Parquet support
pyjwt>=2.8.0            # JWT generation
cryptography>=42.0.0    # Certificate handling
httpx>=0.27.0           # HTTP client
rich>=13.0.0            # CLI progress bars
click>=8.1.0            # CLI framework
pydantic>=2.6.0         # Data validation
```

Install: `pip install -r requirements.txt`

---

## License

MIT License. See [LICENSE](LICENSE) file.
Copyright (c) 2024-2026 Jag Valaiyapathy