mirror of
https://github.com/trailofbits/buttercup
synced 2026-06-21 14:11:39 +00:00
docs(skills): add Langfuse agent skills (#545)
- .claude/skills/langfuse/: vendored from github.com/langfuse/skills (MIT). Covers platform-level usage: langfuse-cli, docs retrieval via llms.txt, instrumentation, prompt migration, SDK upgrades, error analysis, user-feedback scoring. - .claude/skills/buttercup-langfuse/: project-specific wiring — common.llm.get_langfuse_callbacks(), the canonical RunnableConfig pattern used in patcher and seed-gen, and the Helm/values surface for k8s deployment. Defers platform questions to the upstream langfuse skill. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
Riccardo Schirone
committed by
GitHub
GitHub
parent
761dc6c13f
commit
f3e9934f1d
@@ -0,0 +1,109 @@
|
||||
---
|
||||
name: buttercup-langfuse
|
||||
description: Buttercup-specific Langfuse wiring. Use when adding LLM tracing to a new LangChain/LangGraph entry point in this repo, attaching tags/metadata to existing chains, troubleshooting "no traces in Langfuse" against this codebase, or touching the Helm/values surface (`global.langfuse.*`, `langfuse-secret.yaml`, `common-env.yaml`). For platform-level Langfuse usage (CLI, instrumentation patterns generally, prompt migration, SDK upgrades), use the `langfuse` skill instead.
|
||||
---
|
||||
|
||||
# Langfuse wiring in Buttercup
|
||||
|
||||
How *this repo* integrates Langfuse. For general Langfuse platform/CLI/instrumentation guidance, defer to the `langfuse` skill — this skill only covers Buttercup-local conventions.
|
||||
|
||||
Tracing is always optional at runtime: if the env vars are unset or the host is unreachable, callbacks resolve to an empty list and code paths are unaffected. Follow the existing pattern — do not introduce a parallel integration.
|
||||
|
||||
## The one helper to use
|
||||
|
||||
`buttercup.common.llm.get_langfuse_callbacks() -> list[BaseCallbackHandler]`
|
||||
|
||||
- Defined in `common/src/buttercup/common/llm.py`
|
||||
- `@functools.cache`d — cheap to call repeatedly, but the auth/connectivity probe runs only once per process
|
||||
- Returns `[]` when Langfuse is disabled, misconfigured, or unreachable
|
||||
- Internally: runs `is_langfuse_available()` (checks `LANGFUSE_HOST`, then HTTP-probes `/api/public/ingestion`), constructs `langfuse.langchain.CallbackHandler()`, then verifies credentials via `langfuse_auth_check()`
|
||||
|
||||
Do **not** instantiate `CallbackHandler` directly or read the env vars yourself. Always go through `get_langfuse_callbacks()`.
|
||||
|
||||
## Standard wiring pattern
|
||||
|
||||
Attach the callbacks via `RunnableConfig` on the compiled chain/graph. Canonical examples in the repo:
|
||||
|
||||
- `patcher/src/buttercup/patcher/agents/leader.py` — LangGraph with tags + metadata
|
||||
- `seed-gen/src/buttercup/seed_gen/seed_explore.py` — minimal LangGraph
|
||||
- `seed-gen/src/buttercup/seed_gen/task.py`, `vuln_base_task.py`, `seed_init.py` — variants
|
||||
|
||||
Minimal form:
|
||||
|
||||
```python
|
||||
from langchain_core.runnables import RunnableConfig
|
||||
from buttercup.common.llm import get_langfuse_callbacks
|
||||
|
||||
llm_callbacks = get_langfuse_callbacks()
|
||||
chain = workflow.compile().with_config(
|
||||
RunnableConfig(tags=["<short-task-name>"], callbacks=llm_callbacks),
|
||||
)
|
||||
chain.invoke(state)
|
||||
```
|
||||
|
||||
Rich form (patcher-style) — `tags` for filtering, `metadata` for searchable fields in the Langfuse UI:
|
||||
|
||||
```python
|
||||
chain = patch_team.compile().with_config(
|
||||
RunnableConfig(
|
||||
callbacks=llm_callbacks,
|
||||
tags=["patch_team", challenge.name, task_id, internal_patch_id],
|
||||
metadata={
|
||||
"task_id": task_id,
|
||||
"internal_patch_id": internal_patch_id,
|
||||
"challenge_project_name": challenge.name,
|
||||
},
|
||||
recursion_limit=RECURSION_LIMIT,
|
||||
configurable={...},
|
||||
),
|
||||
)
|
||||
```
|
||||
|
||||
Conventions:
|
||||
- First tag is the workflow/team name (`"patch_team"`, `"seed-explore"`, …). Identifies the *kind* of run in the Langfuse UI.
|
||||
- Subsequent tags are high-cardinality identifiers (task_id, challenge name, patch id) for drill-down.
|
||||
- `metadata` mirrors the identifier tags as structured fields — tags are best for filtering, metadata for inspection.
|
||||
- Don't reach for `langfuse.observe()` decorators or other SDK entry points — this codebase routes everything through the LangChain callback handler.
|
||||
|
||||
## When you're adding a NEW LangChain/LangGraph entry point
|
||||
|
||||
1. Import `get_langfuse_callbacks` from `buttercup.common.llm`.
|
||||
2. Call it once near where you compile the chain/graph.
|
||||
3. Pass through `RunnableConfig(callbacks=..., tags=[...], metadata={...})`.
|
||||
4. Pick a workflow tag that doesn't collide with existing ones (`patch_team`, `seed-explore`, `seed-init`, `vuln-discovery`, …).
|
||||
5. If the component runs in k8s, confirm its Deployment template pulls in the Langfuse env block (see below). Most existing components already do.
|
||||
|
||||
## Deployment / config surface
|
||||
|
||||
Env vars (consumed by `get_langfuse_callbacks()`):
|
||||
- `LANGFUSE_HOST` — base URL, e.g. `https://cloud.langfuse.com`
|
||||
- `LANGFUSE_PUBLIC_KEY` (`pk-lf-...`)
|
||||
- `LANGFUSE_SECRET_KEY` (`sk-lf-...`)
|
||||
|
||||
Where they're set:
|
||||
- Local docker-compose: `dev/docker-compose/env.template`
|
||||
- Local make-based deploy: `deployment/env.template` (gated by `LANGFUSE_ENABLED`)
|
||||
- Kubernetes: `deployment/k8s/values.yaml` under `global.langfuse.{enabled,host,publicKey,secretKey}`
|
||||
- Rendered into the `<release>-langfuse-secrets` Secret by `deployment/k8s/templates/langfuse-secret.yaml`
|
||||
- Injected into pods via the `buttercup.env.langfuse` helper in `deployment/k8s/templates/common-env.yaml`
|
||||
- A new component's Deployment template must include that helper to get traces
|
||||
|
||||
Python dependency: `langfuse ~=4.0.1`, declared under `[project.optional-dependencies] full` in `common/pyproject.toml`. Provided via the `[full]` extra of `common`. Components that emit traces depend on `common[full]` (as patcher and seed-gen already do). Don't pin `langfuse` independently — re-use the extra.
|
||||
|
||||
## Debugging "I don't see traces"
|
||||
|
||||
Walk the chain top-to-bottom; the first failing step short-circuits everything downstream:
|
||||
|
||||
1. **`LANGFUSE_HOST` unset.** Logs: `"LangFuse not configured"`. Set the env var.
|
||||
2. **Host unreachable / not Langfuse.** `is_langfuse_available()` POSTs to `/api/public/ingestion` and expects HTTP 401 (unauthenticated). Anything else → `False`. Check network reachability from the pod and that the URL is correct.
|
||||
3. **Bad keys.** Logs: `"LangFuse authentication failed"`. `langfuse_auth_check()` POSTs the same endpoint with basic auth and expects HTTP 400 (authenticated but bad payload). 401 means wrong keys. Rotate the secret and re-apply.
|
||||
4. **Callbacks not attached.** `get_langfuse_callbacks()` returned a handler, but the runnable was invoked without `RunnableConfig(callbacks=...)`. Grep the call site — easy to miss when refactoring.
|
||||
5. **Wrong runnable.** Only LangChain/LangGraph runnables route through callbacks. Direct HTTP calls to OpenAI/LiteLLM bypass Langfuse entirely. If the new code path uses `httpx.post(...)` or similar, it won't trace — that's expected.
|
||||
6. **Cache staleness.** All three helpers are `@functools.cache`d per process. If env vars change at runtime, the process must restart. In k8s this means rolling the pod after a secret change.
|
||||
|
||||
## What NOT to do
|
||||
|
||||
- Don't add a new `langfuse` direct dependency in a sibling component's `pyproject.toml` — depend on `common[full]` instead (as patcher and seed-gen do) to pull it in via the extra.
|
||||
- Don't gate code paths on `is_langfuse_available()` — the empty-callback-list pattern already makes Langfuse a no-op when disabled.
|
||||
- Don't pass the callback list into individual `llm.invoke()` calls when there's a parent chain — attach at the highest reasonable scope (the compiled workflow) so child runs nest correctly in the Langfuse UI.
|
||||
- Don't conflate Langfuse with OpenTelemetry. They coexist: Langfuse for LLM trace nesting/prompt inspection, OTel (`buttercup.common.telemetry`) for system-level spans. `seed_explore.py` shows both wrapped around the same `chain.invoke(...)`.
|
||||
@@ -0,0 +1,140 @@
|
||||
---
|
||||
name: langfuse
|
||||
description: Interact with Langfuse and access its documentation. Use when needing to (1) query or modify Langfuse data programmatically via the CLI — traces, prompts, datasets, scores, sessions, and any other API resource, (2) look up Langfuse documentation, concepts, integration guides, or SDK usage, or (3) understand how any Langfuse feature works. This skill covers CLI-based API access (via npx) and multiple documentation retrieval methods.
|
||||
allowed-tools:
|
||||
- WebFetch(domain:langfuse.com)
|
||||
- Bash(curl *langfuse.com/*)
|
||||
- Bash(npx langfuse-cli api __schema *)
|
||||
- Bash(npx langfuse-cli api * --help *)
|
||||
- Bash(npx langfuse-cli api * list *)
|
||||
- Bash(npx langfuse-cli api * get *)
|
||||
- Bash(bunx langfuse-cli api __schema *)
|
||||
- Bash(bunx langfuse-cli api * --help *)
|
||||
- Bash(bunx langfuse-cli api * list *)
|
||||
- Bash(bunx langfuse-cli api * get *)
|
||||
---
|
||||
|
||||
# Langfuse
|
||||
|
||||
This skill helps you use Langfuse effectively across all common workflows: instrumenting applications, migrating prompts, debugging traces, and accessing data programmatically.
|
||||
|
||||
## Core Principles
|
||||
|
||||
Follow these principles for ALL Langfuse work:
|
||||
|
||||
1. **Documentation First**: NEVER implement based on memory. Always fetch current docs before writing code (Langfuse updates frequently) See the section below on how to access documentation.
|
||||
2. **CLI for Data Access**: Use `langfuse-cli` when querying/modifying Langfuse data. See the section below on how to use the CLI.
|
||||
3. **Best Practices by Use Case**: Check the relevant reference file below for use-case-specific guidelines before implementing
|
||||
4. **Use latest Langfuse versions**: Unless the user specified otherwise or there's a good reason, always use the latest version of Langfuse SDKs/APIs.
|
||||
|
||||
|
||||
## Use case specific references
|
||||
|
||||
- instrumenting an existing function/application: references/instrumentation.md
|
||||
- migrating prompts from a codebase into Langfuse: references/prompt-migration.md
|
||||
- capturing user feedback (thumbs, ratings, implicit signals) as scores on traces: references/user-feedback.md
|
||||
- further tips on using the Langfuse CLI: references/cli.md
|
||||
- upgrading or migrating Langfuse SDKs to the latest version: references/sdk-upgrade.md
|
||||
- systematic error analysis — reading traces, building failure taxonomy, deciding what to fix: references/error-analysis.md
|
||||
- submitting feedback about this skill: references/skill-feedback.md
|
||||
|
||||
## 1. Langfuse API via CLI
|
||||
|
||||
Use the `langfuse-cli` to interact with the full Langfuse REST API from the command line. Run via npx (no install required):
|
||||
|
||||
Start by discovering the schema and available arguments:
|
||||
|
||||
```bash
|
||||
# Discover all available resources
|
||||
npx langfuse-cli api __schema
|
||||
|
||||
# List actions for a resource
|
||||
npx langfuse-cli api <resource> --help
|
||||
|
||||
# Show args/options for a specific action
|
||||
npx langfuse-cli api <resource> <action> --help
|
||||
```
|
||||
|
||||
### Credentials
|
||||
|
||||
Set environment variables before making calls:
|
||||
|
||||
```bash
|
||||
export LANGFUSE_PUBLIC_KEY=pk-lf-...
|
||||
export LANGFUSE_SECRET_KEY=sk-lf-...
|
||||
export LANGFUSE_HOST=https://cloud.langfuse.com # example for EU cloud. For US cloud it's us.cloud.langfuse.com, and can also be a self-hosted URL. The server must always be specified in order to access Langfuse.
|
||||
```
|
||||
|
||||
If not set, ask the user to set them in their shell or a `.env` file (do not ask them to paste keys into chat for security reasons). Keys are found in Langfuse UI → Settings → API Keys.
|
||||
|
||||
### Detailed CLI Reference
|
||||
|
||||
For common workflows, tips, and full usage patterns, see [references/cli.md](references/cli.md).
|
||||
|
||||
## 2. Langfuse Documentation
|
||||
|
||||
Three methods to access Langfuse docs, in order of preference. **Always prefer your application's native web fetch and search tools** (e.g., `WebFetch`, `WebSearch`, `mcp_fetch`, etc.) over `curl` when available. The URLs and patterns below work with any fetching method — the `curl` examples are just illustrative.
|
||||
|
||||
### 2a. Documentation Index (llms.txt)
|
||||
|
||||
Fetch the full index of all documentation pages:
|
||||
|
||||
```bash
|
||||
curl -s https://langfuse.com/llms.txt
|
||||
```
|
||||
|
||||
Returns a structured list of every doc page with titles and URLs. Use this to discover the right page for a topic, then fetch that page directly.
|
||||
|
||||
Alternatively, you can start on `https://langfuse.com/docs` and explore the site to find the page you need.
|
||||
|
||||
### 2b. Fetch Individual Pages as Markdown
|
||||
|
||||
Any page listed in llms.txt can be fetched as markdown by appending `.md` to its path or by using `Accept: text/markdown` in the request headers. Use this when you know which page contains the information needed. Returns clean markdown with code examples and configuration details.
|
||||
|
||||
```bash
|
||||
curl -s "https://langfuse.com/docs/observability/overview.md"
|
||||
curl -s "https://langfuse.com/docs/observability/overview" -H "Accept: text/markdown"
|
||||
```
|
||||
|
||||
### 2c. Search Documentation
|
||||
|
||||
When you need to find information across all docs and github issues/discussions without knowing the specific page:
|
||||
|
||||
```bash
|
||||
curl -s "https://langfuse.com/api/search-docs?query=<url-encoded-query>"
|
||||
```
|
||||
|
||||
Example:
|
||||
|
||||
```bash
|
||||
curl -s "https://langfuse.com/api/search-docs?query=How+do+I+trace+LangGraph+agents"
|
||||
```
|
||||
|
||||
Returns a JSON response with:
|
||||
|
||||
- `query`: the original query
|
||||
- `answer`: a JSON string containing an array of matching documents, each with:
|
||||
- `url`: link to the doc page
|
||||
- `title`: page title
|
||||
- `source.content`: array of relevant text excerpts from the page
|
||||
|
||||
Search is a great fallback if you cannot find the relevant pages or need more context. Especially useful when debugging issues as all GitHub Issues and Discussions are also indexed. Responses can be large — extract only the relevant portions.
|
||||
|
||||
### Documentation Workflow
|
||||
|
||||
1. Start with **llms.txt** to orient — scan for relevant page titles
|
||||
2. **Fetch specific pages** when you identify the right one
|
||||
3. Fall back to **search** when the topic is unclear and you want more context
|
||||
|
||||
## Skill Feedback
|
||||
|
||||
When the user expresses that something about this skill is not working as expected, gives incorrect guidance, is missing information, or could be improved — offer to submit feedback to the Langfuse skill maintainers. This includes when:
|
||||
|
||||
- The skill gave wrong or outdated instructions
|
||||
- A workflow didn't produce the expected result
|
||||
- The user wishes the skill covered something it doesn't
|
||||
- The user explicitly says something like "this should work differently" or "this is wrong"
|
||||
|
||||
**Do NOT trigger this** for issues with Langfuse itself (the product) — only for issues with this skill's instructions and behavior.
|
||||
|
||||
When triggered, follow the process in [references/skill-feedback.md](references/skill-feedback.md).
|
||||
@@ -0,0 +1,51 @@
|
||||
# Langfuse CLI Reference
|
||||
|
||||
Documentation: https://langfuse.com/docs/api-and-data-platform/features/cli
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
# Run directly (recommended)
|
||||
npx langfuse-cli api <resource> <action>
|
||||
bunx langfuse-cli api <resource> <action>
|
||||
|
||||
# Or install globally
|
||||
npm i -g langfuse-cli
|
||||
langfuse api <resource> <action>
|
||||
```
|
||||
|
||||
## Discovery
|
||||
|
||||
```bash
|
||||
# List all resources and auth info
|
||||
langfuse api __schema
|
||||
|
||||
# List actions for a resource
|
||||
langfuse api <resource> --help
|
||||
|
||||
# Show args/options for a specific action
|
||||
langfuse api <resource> <action> --help
|
||||
|
||||
# Preview the curl command without executing
|
||||
langfuse api <resource> <action> --curl
|
||||
```
|
||||
|
||||
## Credentials
|
||||
|
||||
Set environment variables:
|
||||
|
||||
```bash
|
||||
export LANGFUSE_PUBLIC_KEY=pk-lf-...
|
||||
export LANGFUSE_SECRET_KEY=sk-lf-...
|
||||
export LANGFUSE_HOST=https://cloud.langfuse.com
|
||||
```
|
||||
|
||||
## Tips
|
||||
|
||||
- Use `--json` for machine-readable JSON output
|
||||
- Use `--curl` to preview the HTTP request without executing
|
||||
- Pagination: use `--limit` and `--page` on list endpoints
|
||||
- All list commands support filtering — check `<resource> <action> --help` for available options
|
||||
- Prefer `observations-v2s` over `observations` — the v2 endpoint returns richer data
|
||||
- Prefer `metrics-v2s` over `metrics` — the v2 endpoint returns richer data
|
||||
- Prefer `score-v2s` over `scores` — the v1 `scores` resource only supports create/delete; use `score-v2s` for list and get operations
|
||||
@@ -0,0 +1,100 @@
|
||||
---
|
||||
name: langfuse-error-analysis
|
||||
description: Deep-dive error analysis of an LLM pipeline or AI application using Langfuse traces.
|
||||
Use this skill whenever the user wants to understand why their AI system is producing
|
||||
bad outputs, where their pipeline is failing, how to categorise or label failures,
|
||||
what to prioritise fixing, or how to set up evaluators. Also trigger for "review my
|
||||
traces", "my outputs look wrong", "help me debug my LLM app", "I want to analyse
|
||||
errors", "build a failure taxonomy", "what's going wrong with my pipeline", or any
|
||||
request to systematically inspect, annotate, or score Langfuse traces. If the user
|
||||
is trying to understand or improve the quality of an AI system's outputs, use this skill.
|
||||
---
|
||||
|
||||
# Error Analysis
|
||||
|
||||
## Primary Guide
|
||||
|
||||
**1. Fetch the guide in this blogpost**
|
||||
|
||||
https://langfuse.com/guides/cookbook/error-analysis-llm-applications.md
|
||||
|
||||
If fetch is not available query for langfuse.com error analysis guide
|
||||
|
||||
Read it in full. It defines the authoritative 5-step process (sample selection → open coding → clustering → labelling → deciding what to fix).
|
||||
|
||||
**2. Guide the user through this step by step**
|
||||
|
||||
You as a coding agent and the user go through this together to perform a full error analysis with their data in langfuse. Do everything you can achieve via CLI (look up traces, create annotation queues, ...) for the user. Provide them with direct links to UI wherever their action is required. Be proactive and narrate what is going on for the user.
|
||||
|
||||
## Rules CRITICAL
|
||||
Use Langfuse CLI wherever possible
|
||||
Use charts where possible to display data
|
||||
|
||||
---
|
||||
|
||||
## Langfuse Implementation Notes
|
||||
|
||||
The guide describes the process. These notes cover the Langfuse-specific API and CLI mechanics required to execute it.
|
||||
|
||||
### Credentials
|
||||
|
||||
```bash
|
||||
echo $LANGFUSE_PUBLIC_KEY # pk-lf-...
|
||||
echo $LANGFUSE_SECRET_KEY # sk-lf-...
|
||||
echo $LANGFUSE_HOST # https://cloud.langfuse.com (EU), https://us.cloud.langfuse.com (US), https://jp.cloud.langfuse.com (JP) or self-hosted
|
||||
```
|
||||
|
||||
If not set, check `.env` in the project root: `export $(grep -v '^#' .env | xargs)`. If `LANGFUSE_BASE_URL` is used instead of `LANGFUSE_HOST`, run `export LANGFUSE_HOST="$LANGFUSE_BASE_URL"`.
|
||||
|
||||
```bash
|
||||
AUTH=$(echo -n "${LANGFUSE_PUBLIC_KEY}:${LANGFUSE_SECRET_KEY}" | base64)
|
||||
|
||||
# Verify before proceeding
|
||||
STATUS=$(curl -s -o /dev/null -w "%{http_code}" \
|
||||
-H "Authorization: Basic $AUTH" \
|
||||
"${LANGFUSE_HOST}/api/public/projects")
|
||||
echo "Auth check: $STATUS"
|
||||
```
|
||||
|
||||
If status is not `200`, stop and ask the user to check their credentials and host before continuing.
|
||||
|
||||
### Annotation target: OBSERVATION versus TRACE
|
||||
|
||||
> **CRITICAL:** In OpenTelemetry-instrumented apps, trace-level `input`/`output` can be null — content often lives in a GENERATION observation. Always consider if the right objectType to add is `objectType: OBSERVATION` pointing to the GENERATION observation ID to annotation queues.
|
||||
|
||||
### Annotation queues
|
||||
|
||||
> **CRITICAL:** Queues cannot be updated or deleted after creation. Create score configs first, then the queue with all config IDs. To add new configs later, create a new queue.
|
||||
|
||||
|
||||
**Always give the user a direct link immediately after creating a queue:**
|
||||
|
||||
| Host | URL pattern |
|
||||
|------|-------------|
|
||||
| EU cloud | `https://cloud.langfuse.com/project/<projectId>/annotation-queues/<queueId>` |
|
||||
| US cloud | `https://us.cloud.langfuse.com/project/<projectId>/annotation-queues/<queueId>` |
|
||||
| Self-hosted | `<LANGFUSE_HOST>/project/<projectId>/annotation-queues/<queueId>` |
|
||||
|
||||
Instruction to give: *"Please open code the first ~50 examples. For each trace, write what you observe in the `open_coding` field (describe behaviour, don't diagnose root causes), then set `pass_fail_assessment` to Pass or Fail."*
|
||||
|
||||
|
||||
### Prompt fixes
|
||||
|
||||
When a category warrants a prompt fix, always offer the user two options:
|
||||
1. Create it as a versioned prompt in Langfuse (tracked, usable via the prompt API)
|
||||
2. Draft the specific text change for them to review and apply
|
||||
|
||||
### Setup evaluators
|
||||
|
||||
When a category warrants an evaluator setup, propose the type of evaluator and offer to set it up for user via CLI
|
||||
|
||||
|
||||
### Common gotchas
|
||||
|
||||
| Mistake | Fix |
|
||||
|---------|-----|
|
||||
| `objectType: TRACE` in queue | Use `objectType: OBSERVATION` with GENERATION obs ID |
|
||||
| Creating score config without checking existing | `GET /api/public/score-configs` first; can't delete |
|
||||
| Queue created before score configs | Create configs → collect IDs → create queue |
|
||||
| `--limit` > 100 on traces list | API hard cap; paginate with `--page` |
|
||||
| No rate limiting on queue item creation | `sleep 0.4` between calls to avoid 429 |
|
||||
@@ -0,0 +1,140 @@
|
||||
---
|
||||
name: langfuse-observability
|
||||
description: Instrument LLM applications with Langfuse tracing. Use when setting up Langfuse, adding observability to LLM calls, or auditing existing instrumentation.
|
||||
---
|
||||
|
||||
# Langfuse Observability
|
||||
|
||||
Instrument LLM applications with Langfuse tracing, following best practices and tailored to your use case.
|
||||
|
||||
## When to Use
|
||||
|
||||
- Setting up Langfuse in a new project
|
||||
- Auditing existing Langfuse instrumentation
|
||||
- Adding observability to LLM calls
|
||||
|
||||
## Workflow
|
||||
|
||||
### 1. Assess Current State
|
||||
|
||||
Check the project:
|
||||
|
||||
- Is Langfuse SDK installed?
|
||||
- What LLM frameworks are used? (OpenAI SDK, LangChain, LlamaIndex, Vercel AI SDK, etc.)
|
||||
- Is there existing instrumentation?
|
||||
|
||||
**No integration yet:** Set up Langfuse using a framework integration if available. Integrations capture more context automatically and require less code than manual instrumentation.
|
||||
|
||||
**Integration exists:** Audit against baseline requirements below.
|
||||
|
||||
### 2. Verify Baseline Requirements
|
||||
|
||||
Every trace should have these fundamentals:
|
||||
|
||||
| Requirement | Check | Why |
|
||||
| ------------------------- | ---------------------------------------------------------------------------------------- | ------------------------------------------------------ |
|
||||
| Model name | Is the LLM model captured? | Enables model comparison and filtering |
|
||||
| Token usage | Are input/output tokens tracked? | Enables automatic cost calculation |
|
||||
| Good trace names | Are names descriptive? (`chat-response`, not `trace-1`) | Makes traces findable and filterable |
|
||||
| Span hierarchy | Are multi-step operations nested properly? | Shows which step is slow or failing |
|
||||
| Correct observation types | Are generations marked as generations? | Enables model-specific analytics |
|
||||
| Sensitive data masked | Is PII/confidential data excluded or masked? | Prevents data leakage |
|
||||
| Trace input/output | Does the trace capture meaningful input/output? Is input explicitly set to show only relevant data (e.g., user message), not all function args? | Makes traces readable in the UI and avoids leaking sensitive args |
|
||||
|
||||
Framework integrations (OpenAI, LangChain, etc.) handle model name, tokens, and observation types automatically. Prefer integrations over manual instrumentation.
|
||||
|
||||
Docs: https://langfuse.com/docs/tracing
|
||||
|
||||
### 3. Explore Traces First
|
||||
|
||||
Once baseline instrumentation is working, encourage the user to explore their traces in the Langfuse UI before adding more context:
|
||||
|
||||
"Your traces are now appearing in Langfuse. Take a look at a few of them—see what data is being captured, what's useful, and what's missing. This will help us decide what additional context to add."
|
||||
|
||||
This helps the user:
|
||||
|
||||
- Understand what they're already getting
|
||||
- Form opinions about what's missing
|
||||
- Ask better questions about what they need
|
||||
|
||||
### 4. Discover Additional Context Needs
|
||||
|
||||
Determine what additional instrumentation would be valuable. **Infer from code when possible, only ask when unclear.**
|
||||
|
||||
**Infer from code:**
|
||||
|
||||
| If you see in code... | Infer | Suggest |
|
||||
| ---------------------------------------------------- | ----------------- | ------------------------- |
|
||||
| Conversation history, chat endpoints, message arrays | Multi-turn app | `session_id` |
|
||||
| User authentication, `user_id` variables | User-aware app | `user_id` on traces |
|
||||
| Multiple distinct endpoints/features | Multi-feature app | `feature` tag |
|
||||
| Customer/tenant identifiers | Multi-tenant app | `customer_id` or tier tag |
|
||||
| Feedback collection, ratings | Has user feedback | Capture as scores |
|
||||
|
||||
**Only ask when not obvious from code:**
|
||||
|
||||
- "How do you know when a response is good vs bad?" → Determines scoring approach
|
||||
- "What would you want to filter by in a dashboard?" → Surfaces non-obvious tags
|
||||
- "Are there different user segments you'd want to compare?" → Customer tiers, plans, etc.
|
||||
|
||||
**Additions and their value:**
|
||||
|
||||
| Addition | Why | Docs |
|
||||
| ------------------- | ------------------------------------------- | --------------------------------------------------- |
|
||||
| `session_id` | Groups conversations together | https://langfuse.com/docs/tracing-features/sessions |
|
||||
| `user_id` | Enables user filtering and cost attribution | https://langfuse.com/docs/tracing-features/users |
|
||||
| User feedback score | Enables quality filtering and trends | https://langfuse.com/docs/scores/overview |
|
||||
| `feature` tag | Per-feature analytics | https://langfuse.com/docs/tracing-features/tags |
|
||||
| `customer_tier` tag | Cost/quality breakdown by segment | https://langfuse.com/docs/tracing-features/tags |
|
||||
|
||||
These are NOT baseline requirements—only add what's relevant based on inference or user input.
|
||||
|
||||
### 5. Guide to UI
|
||||
|
||||
After adding context, point users to relevant UI features:
|
||||
|
||||
- Traces view: See individual requests
|
||||
- Sessions view: See grouped conversations (if session_id added)
|
||||
- Dashboard: Build filtered views using tags
|
||||
- Scores: Filter by quality metrics
|
||||
|
||||
## Framework Integrations
|
||||
|
||||
Prefer these over manual instrumentation:
|
||||
|
||||
| Framework | Integration | Docs |
|
||||
| ------------- | ---------------------- | ---------------------------------------------------- |
|
||||
| OpenAI SDK | Drop-in replacement | https://langfuse.com/docs/integrations/openai |
|
||||
| LangChain | Callback handler | https://langfuse.com/docs/integrations/langchain |
|
||||
| LlamaIndex | Callback handler | https://langfuse.com/docs/integrations/llama-index |
|
||||
| Vercel AI SDK | OpenTelemetry exporter | https://langfuse.com/docs/integrations/vercel-ai-sdk |
|
||||
| LiteLLM | Callback or proxy | https://langfuse.com/docs/integrations/litellm |
|
||||
|
||||
Full list: https://langfuse.com/docs/integrations
|
||||
|
||||
## Always Explain Why
|
||||
|
||||
When suggesting additions, explain the user benefit:
|
||||
|
||||
```
|
||||
"I recommend adding session_id to your traces.
|
||||
|
||||
Why: This groups messages from the same conversation together.
|
||||
You'll be able to see full conversation flows in the Sessions view,
|
||||
making it much easier to debug multi-turn interactions.
|
||||
|
||||
Learn more: https://langfuse.com/docs/tracing-features/sessions"
|
||||
```
|
||||
|
||||
## Common Mistakes
|
||||
|
||||
| Mistake | Problem | Fix |
|
||||
| ---------------------------------------------- | --------------------------------------------------- | --------------------------------------------------------------------------------- |
|
||||
| No `flush()` in scripts | Traces never sent | Call `langfuse.flush()` before exit |
|
||||
| Flat traces | Can't see which step failed | Use nested spans for distinct steps |
|
||||
| Generic trace names | Hard to filter | Use descriptive names: `chat-response`, `doc-summary` |
|
||||
| Logging sensitive data | Data leakage risk | Mask PII before tracing |
|
||||
| Not explicitly setting input with `@observe` | All function args become trace input (including API keys, configs) | Python: use `langfuse.update_current_span(input=...)`. JS/TS: use `updateActiveObservation({ input: ... })`. Set only the relevant input (e.g., user message) |
|
||||
| Manual instrumentation when integration exists | More code, less context | Use framework integration |
|
||||
| Langfuse import before env vars loaded | Langfuse initializes with missing/wrong credentials | Import Langfuse AFTER loading environment variables (e.g., after `load_dotenv()`) |
|
||||
| Wrong import order with OpenAI | Langfuse can't patch the OpenAI client | Import Langfuse and call its setup BEFORE importing OpenAI client |
|
||||
@@ -0,0 +1,234 @@
|
||||
---
|
||||
name: langfuse-prompt-migration
|
||||
description: Migrate hardcoded prompts to Langfuse for version control and deployment-free iteration. Use when user wants to externalize prompts, move prompts to Langfuse, or set up prompt management.
|
||||
---
|
||||
|
||||
# Langfuse Prompt Migration
|
||||
|
||||
Migrate hardcoded prompts to Langfuse for version control, A/B testing, and deployment-free iteration.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Verify credentials are set before starting. Check existence only — never print the secret key, since the value would land in the agent's context and transcripts:
|
||||
|
||||
```bash
|
||||
[ -n "$LANGFUSE_PUBLIC_KEY" ] && echo "LANGFUSE_PUBLIC_KEY: set" || echo "LANGFUSE_PUBLIC_KEY: missing"
|
||||
[ -n "$LANGFUSE_SECRET_KEY" ] && echo "LANGFUSE_SECRET_KEY: set" || echo "LANGFUSE_SECRET_KEY: missing"
|
||||
[ -n "$LANGFUSE_HOST" ] && echo "LANGFUSE_HOST: $LANGFUSE_HOST" || echo "LANGFUSE_HOST: missing"
|
||||
```
|
||||
|
||||
If not set, ask the user to configure them in their shell or a `.env` file. Do not ask them to paste keys into chat.
|
||||
|
||||
## Migration Flow
|
||||
|
||||
```
|
||||
1. Scan codebase for prompts
|
||||
2. Analyze templating compatibility
|
||||
3. Propose structure (names, subprompts, variables)
|
||||
4. User approves
|
||||
5. Create prompts in Langfuse
|
||||
6. Refactor code to use get_prompt()
|
||||
7. Link prompts to traces (if tracing enabled)
|
||||
8. Verify application works
|
||||
```
|
||||
|
||||
## Step 1: Find Prompts and Build an Inventory
|
||||
|
||||
Before writing ANY code, make a complete list of every prompt you found. For each one, note:
|
||||
|
||||
- Name: descriptive, lowercase, hyphenated (e.g. chat-assistant, email-classifier)
|
||||
- Source file: where the prompt text lives
|
||||
- Code file to refactor: the Python/JS file that USES the prompt (for asset files like .txt/.yaml/.md, this is the file that reads/loads the asset — NOT the asset file itself)
|
||||
- Type: chat (used as a message in a chat API) or text (used as a plain string)
|
||||
- Variables: values interpolated into the prompt, converted to {{var}} syntax:
|
||||
f-string {var} → {{var}}
|
||||
.format(var=...) → {{var}}
|
||||
${var} → {{var}}
|
||||
String concatenation + var + → {{var}}
|
||||
YAML {var} → {{var}}
|
||||
- Prompt content: the actual text to upload, with variables converted to {{var}} syntax
|
||||
|
||||
Search for these patterns:
|
||||
|
||||
| Framework | Look for |
|
||||
|-----------|----------|
|
||||
| OpenAI | `messages=[{"role": "system", "content": "..."}]` |
|
||||
| Anthropic | `system="..."` |
|
||||
| LangChain | `ChatPromptTemplate`, `SystemMessage` |
|
||||
| Vercel AI | `system: "..."`, `prompt: "..."` |
|
||||
| Raw | Multi-line strings near LLM calls |
|
||||
|
||||
## Step 2: Check Templating Compatibility
|
||||
|
||||
**CRITICAL:** Langfuse only supports simple `{{variable}}` substitution. No conditionals, loops, or filters.
|
||||
|
||||
| Template Feature | Langfuse Native | Action |
|
||||
|------------------|-----------------|--------|
|
||||
| `{{variable}}` | ✅ | Direct migration |
|
||||
| `{var}` / `${var}` | ⚠️ | Convert to `{{var}}` |
|
||||
| `{% if %}` / `{% for %}` | ❌ | Move logic to code |
|
||||
| `{{ var \| filter }}` | ❌ | Apply filter in code |
|
||||
|
||||
**CRITICAL — Variable syntax:** Langfuse uses DOUBLE curly braces for variables: `{{var}}`. When uploading prompt content, you MUST convert every single-brace `{var}` from the original code to double-brace `{{var}}`. Never upload `{var}` — it must be `{{var}}`.
|
||||
|
||||
### Decision Tree
|
||||
|
||||
```
|
||||
Contains {% if %}, {% for %}, or filters?
|
||||
├─ No → Direct migration
|
||||
└─ Yes → Choose:
|
||||
├─ Option A (RECOMMENDED): Move logic to code, pass pre-computed values
|
||||
└─ Option B: Store raw template, compile client-side with Jinja2
|
||||
└─ ⚠️ Loses: Playground preview, UI experiments
|
||||
```
|
||||
|
||||
### Simplifying Complex Templates
|
||||
|
||||
**Conditionals** → Pre-compute in code:
|
||||
```python
|
||||
# Instead of {% if user.is_premium %}...{% endif %} in prompt
|
||||
# Use {{tier_message}} and compute value in code before compile()
|
||||
```
|
||||
|
||||
**Loops** → Pre-format in code:
|
||||
```python
|
||||
# Instead of {% for tool in tools %}...{% endfor %} in prompt
|
||||
# Use {{tools_list}} and format the list in code before compile()
|
||||
```
|
||||
|
||||
For external templating details, fetch: https://langfuse.com/faq/all/using-external-templating-libraries
|
||||
|
||||
## Step 3: Propose Structure
|
||||
|
||||
### Naming Conventions
|
||||
|
||||
| Rule | Example | Bad |
|
||||
|------|---------|-----|
|
||||
| Lowercase, hyphenated | `chat-assistant` | `ChatAssistant_v2` |
|
||||
| Feature-based | `document-summarizer` | `prompt1` |
|
||||
| Hierarchical for related | `support/triage` | `supportTriage` |
|
||||
| Prefix subprompts with `_` | `_base-personality` | `shared-personality` |
|
||||
|
||||
### Identify Subprompts
|
||||
|
||||
Extract when:
|
||||
- Same text in 2+ prompts
|
||||
- Represents distinct component (personality, safety rules, format)
|
||||
- Would need to change together
|
||||
|
||||
### Variable Extraction
|
||||
|
||||
| Make Variable | Keep Hardcoded |
|
||||
|---------------|----------------|
|
||||
| User-specific (`{{user_name}}`) | Output format instructions |
|
||||
| Dynamic content (`{{context}}`) | Safety guardrails |
|
||||
| Per-request (`{{query}}`) | Persona/personality |
|
||||
| Environment-specific (`{{company_name}}`) | Static examples |
|
||||
|
||||
## Step 4: Present Plan to User
|
||||
|
||||
Format:
|
||||
```
|
||||
Found N prompts across M files:
|
||||
|
||||
src/chat.py:
|
||||
- System prompt (47 lines) → 'chat-assistant'
|
||||
|
||||
src/support/triage.py:
|
||||
- Triage prompt (34 lines) → 'support/triage'
|
||||
⚠️ Contains {% if %} - will simplify
|
||||
|
||||
Subprompts to extract:
|
||||
- '_base-personality' - used by: chat-assistant, support/triage
|
||||
|
||||
Variables to add:
|
||||
- {{user_name}} - hardcoded in 2 prompts
|
||||
|
||||
Proceed?
|
||||
```
|
||||
|
||||
## Step 5: Create Prompts in Langfuse
|
||||
|
||||
Use `langfuse.create_prompt()` with:
|
||||
- `name`: Your chosen name
|
||||
- `prompt`: Template text (or message array for chat type)
|
||||
- `type`: `"text"` or `"chat"`
|
||||
- `labels`: `["production"]` (they're already live)
|
||||
- `config`: Optional model settings
|
||||
|
||||
**Labeling strategy:**
|
||||
- `production` → All migrated prompts
|
||||
- `staging` → Add later for testing
|
||||
- `latest` → Auto-applied by Langfuse
|
||||
|
||||
For full API: fetch https://langfuse.com/docs/prompts/get-started
|
||||
|
||||
## Step 6: Refactor Code
|
||||
|
||||
Replace hardcoded prompts with:
|
||||
|
||||
```python
|
||||
prompt = langfuse.get_prompt("name", label="production")
|
||||
messages = prompt.compile(var1=value1, var2=value2)
|
||||
```
|
||||
|
||||
**Key points:**
|
||||
- Always use `label="production"` (not `latest`) for stability
|
||||
- Call `.compile()` to substitute variables
|
||||
- For chat prompts, result is message array ready for API
|
||||
|
||||
For SDK examples (Python/JS/TS): fetch https://langfuse.com/docs/prompts/get-started
|
||||
|
||||
## Step 7: Link Prompts to Traces
|
||||
|
||||
If codebase uses Langfuse tracing, link prompts so you can see which version produced each response.
|
||||
|
||||
### Detect Existing Tracing
|
||||
|
||||
Look for:
|
||||
- `@observe()` decorators
|
||||
- `langfuse.trace()` calls
|
||||
- `from langfuse.openai import openai` (instrumented client)
|
||||
|
||||
### Link Methods
|
||||
|
||||
| Setup | How to Link |
|
||||
|-------|-------------|
|
||||
| `@observe()` decorator | `langfuse_context.update_current_observation(prompt=prompt)` |
|
||||
| Manual tracing | `trace.generation(prompt=prompt, ...)` |
|
||||
| OpenAI integration | `openai.chat.completions.create(..., langfuse_prompt=prompt)` |
|
||||
|
||||
### Verify in UI
|
||||
|
||||
1. Go to **Traces** → select a trace
|
||||
2. Click on **Generation**
|
||||
3. Check **Prompt** field shows name and version
|
||||
|
||||
For tracing details: fetch https://langfuse.com/docs/prompts/get-started#link-with-langfuse-tracing
|
||||
|
||||
## Step 8: Verify Migration
|
||||
|
||||
### Checklist
|
||||
|
||||
- [ ] All prompts created with `production` label
|
||||
- [ ] Code fetches with `label="production"`
|
||||
- [ ] Variables compile without errors
|
||||
- [ ] Subprompts resolve correctly
|
||||
- [ ] Application behavior unchanged
|
||||
- [ ] Generations show linked prompt in UI (if tracing)
|
||||
|
||||
### Common Issues
|
||||
|
||||
| Issue | Solution |
|
||||
|-------|----------|
|
||||
| `PromptNotFoundError` | Check name spelling |
|
||||
| Variables not replaced | Use `{{var}}` not `{var}`, call `.compile()` |
|
||||
| Subprompt not resolved | Must exist with same label |
|
||||
| Old prompt cached | Restart app |
|
||||
|
||||
## Out of Scope
|
||||
|
||||
- Prompt engineering (writing better prompts)
|
||||
- Evaluation setup
|
||||
- A/B testing workflow
|
||||
- Non-LLM string templates
|
||||
@@ -0,0 +1,181 @@
|
||||
---
|
||||
name: langfuse-sdk-upgrade
|
||||
description: Upgrade Langfuse SDKs from older versions to the latest. Use when migrating Python SDK v2/v3 to v4, or JS/TS SDK v3/v4 to v5.
|
||||
---
|
||||
|
||||
# Langfuse SDK Upgrade Guide
|
||||
|
||||
Assist users in upgrading their Langfuse SDK to the latest version. The Python and JS/TS SDKs share the same architectural changes but differ in syntax.
|
||||
|
||||
## When to Use
|
||||
|
||||
- User asks to upgrade/migrate their Langfuse SDK
|
||||
- User is on an older SDK version and encounters deprecated APIs
|
||||
- User wants to adopt the latest Langfuse features
|
||||
|
||||
## Migration Docs
|
||||
|
||||
Always fetch the latest migration guide before starting — these pages are the source of truth:
|
||||
|
||||
- **Python (v3 → v4):** https://langfuse.com/docs/observability/sdk/upgrade-path/python-v3-to-v4
|
||||
- **JS/TS (v4 → v5):** https://langfuse.com/docs/observability/sdk/upgrade-path/js-v4-to-v5
|
||||
|
||||
Fetch the relevant page as markdown before implementing any changes:
|
||||
|
||||
```bash
|
||||
curl -s "https://langfuse.com/docs/observability/sdk/upgrade-path/python-v3-to-v4.md"
|
||||
curl -s "https://langfuse.com/docs/observability/sdk/upgrade-path/js-v4-to-v5.md"
|
||||
```
|
||||
|
||||
## Upgrade Checklist
|
||||
|
||||
Work through each item in order. Skip items that don't apply to the user's codebase.
|
||||
|
||||
### Both SDKs
|
||||
|
||||
- [ ] **Update the SDK package** to the latest version
|
||||
- [ ] **Audit span filtering**: Non-LLM spans (HTTP, DB, queues) no longer export by default. If the user relied on these, configure a custom `should_export_span` / `shouldExportSpan` filter
|
||||
- [ ] **Replace `update_current_trace()` / `updateActiveTrace()`**: Split into three calls:
|
||||
- `propagate_attributes()` / `propagateAttributes()` for correlating attributes (`user_id`, `session_id`, `tags`, `metadata`, `trace_name`)
|
||||
- `set_current_trace_io()` / `setActiveTraceIO()` for input/output (deprecated — prefer setting I/O on root observation directly)
|
||||
- `set_current_trace_as_public()` / `setActiveTraceAsPublic()` for public flag
|
||||
- [ ] **Replace `.update_trace()` / `.updateTrace()`** on observation objects (same decomposition as above)
|
||||
- [ ] **Update API namespace references**: `observations_v_2` / `observationsV2` → `observations`, `score_v_2` / `scoreV2` → `scores`, `metrics_v_2` / `metricsV2` → `metrics`. Legacy v1 APIs moved to `api.legacy.*`
|
||||
- [ ] **Validate metadata format**: Must be `dict[str, str]` / `Record<string, string>` with values ≤200 characters
|
||||
- [ ] **Move `release` and `environment`** from code parameters to environment variables (`LANGFUSE_RELEASE`, `LANGFUSE_TRACING_ENVIRONMENT`)
|
||||
- [ ] **Enable debug logging** during migration to catch issues (`debug=True` in Python, `LANGFUSE_DEBUG="true"` in JS/TS)
|
||||
- [ ] **Test trace hierarchies** to verify no spans are unexpectedly dropped
|
||||
|
||||
### Python-specific
|
||||
|
||||
- [ ] **Replace `start_span()` / `start_generation()`** with `start_observation()` (use `as_type="generation"` for generations)
|
||||
- [ ] **Replace `start_as_current_span()` / `start_as_current_generation()`** with `start_as_current_observation()`
|
||||
- [ ] **Replace dataset `item.run()`** with `dataset.run_experiment(name=..., task=...)`
|
||||
- [ ] **Remove `CallbackHandler(update_trace=...)`** parameter — use `propagate_attributes()` wrapper instead
|
||||
- [ ] **Upgrade to Pydantic v2** — the SDK now requires it. Use `pydantic.v1` compatibility shim if migrating gradually
|
||||
- [ ] **Update removed types**: `TraceMetadata`, `ObservationParams` removed from `langfuse.types`. Import `MapValue`, `ModelUsage`, `PromptClient` from `langfuse.model`
|
||||
|
||||
### JS/TS-specific
|
||||
|
||||
- [ ] **Update LangChain `CallbackHandler`** — `traceMetadata` now requires string values; internal behavior uses `propagateAttributes()` instead of direct trace updates
|
||||
- [ ] **Update OpenAI integration** — `traceMethod` wrapper now uses `propagateAttributes()` internally; wrap entire execution in `propagateAttributes()` if relying on parent attribute inheritance
|
||||
|
||||
## Key API Changes Reference
|
||||
|
||||
### Correlating attributes (both SDKs)
|
||||
|
||||
**Before:**
|
||||
```python
|
||||
# Python
|
||||
langfuse.update_current_trace(name="trace-name", user_id="user-123", session_id="session-abc", tags=["tag1"])
|
||||
```
|
||||
```typescript
|
||||
// JS/TS
|
||||
updateActiveTrace({ name: "trace-name", userId: "user-123", sessionId: "session-456", tags: ["prod"] });
|
||||
```
|
||||
|
||||
**After:**
|
||||
```python
|
||||
# Python
|
||||
from langfuse import propagate_attributes
|
||||
|
||||
with propagate_attributes(trace_name="trace-name", user_id="user-123", session_id="session-abc", tags=["tag1"]):
|
||||
result = call_llm("hello")
|
||||
```
|
||||
```typescript
|
||||
// JS/TS
|
||||
import { propagateAttributes } from "langfuse";
|
||||
|
||||
await propagateAttributes(
|
||||
{ traceName: "trace-name", userId: "user-123", sessionId: "session-456", tags: ["prod"] },
|
||||
async () => { /* traced code */ }
|
||||
);
|
||||
```
|
||||
|
||||
### Span/Generation creation (Python)
|
||||
|
||||
**Before:**
|
||||
```python
|
||||
langfuse.start_span(name="x")
|
||||
langfuse.start_generation(name="x", model="gpt-4")
|
||||
```
|
||||
|
||||
**After:**
|
||||
```python
|
||||
langfuse.start_observation(name="x")
|
||||
langfuse.start_observation(name="x", as_type="generation", model="gpt-4")
|
||||
```
|
||||
|
||||
### Dataset experiments (Python)
|
||||
|
||||
**Before:**
|
||||
```python
|
||||
for item in dataset.items:
|
||||
with item.run(run_name="my-run") as span:
|
||||
result = my_llm(item.input)
|
||||
span.update(output=result)
|
||||
```
|
||||
|
||||
**After:**
|
||||
```python
|
||||
def my_task(*, item, **kwargs):
|
||||
return my_llm(item.input)
|
||||
|
||||
dataset.run_experiment(name="my-run", task=my_task)
|
||||
```
|
||||
|
||||
### Span filtering (both SDKs)
|
||||
|
||||
To restore pre-upgrade "export all" behavior:
|
||||
|
||||
```python
|
||||
# Python
|
||||
langfuse = Langfuse(should_export_span=lambda span: True)
|
||||
```
|
||||
```typescript
|
||||
// JS/TS
|
||||
const spanProcessor = new LangfuseSpanProcessor({ shouldExportSpan: () => true });
|
||||
```
|
||||
|
||||
To extend defaults with custom scopes:
|
||||
|
||||
```python
|
||||
# Python
|
||||
from langfuse.span_filter import is_default_export_span
|
||||
|
||||
langfuse = Langfuse(
|
||||
should_export_span=lambda span: (
|
||||
is_default_export_span(span)
|
||||
or span.instrumentation_scope.name.startswith("my_framework")
|
||||
)
|
||||
)
|
||||
```
|
||||
```typescript
|
||||
// JS/TS
|
||||
import { isDefaultExportSpan } from "@langfuse/otel";
|
||||
|
||||
shouldExportSpan: ({ otelSpan }) =>
|
||||
isDefaultExportSpan(otelSpan) || otelSpan.instrumentationScope.name.startsWith("my_framework")
|
||||
```
|
||||
|
||||
## Common Pitfalls
|
||||
|
||||
| Pitfall | Impact | Fix |
|
||||
| --- | --- | --- |
|
||||
| Dropping intermediate spans via filtering | Breaks trace trees — child spans become orphaned | Use `is_default_export_span` as base and only add/remove specific scopes |
|
||||
| Metadata with non-string values | Values silently coerced or dropped | Ensure all metadata values are strings ≤200 characters |
|
||||
| Setting attributes outside `propagate_attributes()` callback | Attributes don't attach to observations | Wrap all traced code inside the callback |
|
||||
| Using deprecated `set_current_trace_io()` for new code | Will be removed in future versions | Set input/output directly on the root observation |
|
||||
| Forgetting Pydantic v2 upgrade (Python) | Import errors or runtime failures | Upgrade Pydantic or use `pydantic.v1` shim |
|
||||
| `release`/`environment` still passed as parameters | Silently ignored | Use `LANGFUSE_RELEASE` and `LANGFUSE_TRACING_ENVIRONMENT` env vars |
|
||||
| LangChain/OpenAI attribute propagation direction changed | Attributes propagate downward only, not upward to parent traces | Wrap outer call in `propagate_attributes()` |
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Always fetch the migration docs first** — they are the canonical source and may have been updated since this guide was written
|
||||
2. **Enable debug logging during migration** to surface dropped spans and trace hierarchy issues
|
||||
3. **Use `propagate_attributes()` as the primary mechanism** for setting trace-level correlating attributes
|
||||
4. **Set input/output on root observations directly** rather than using deprecated trace-level setters
|
||||
5. **Compose custom span filters** with `is_default_export_span` / `isDefaultExportSpan` to extend defaults rather than replacing them entirely
|
||||
6. **Test thoroughly** — run the application with debug logging, check the Langfuse UI for missing or orphaned spans, verify metadata appears correctly
|
||||
7. **Migrate incrementally** — upgrade the SDK first, fix breaking changes, then adopt new patterns
|
||||
@@ -0,0 +1,52 @@
|
||||
---
|
||||
name: langfuse-skill-feedback
|
||||
description: Submit feedback about the Langfuse skill to its maintainers via GitHub Discussions. Use when the user indicates the skill gave incorrect guidance, is missing information, or could be improved.
|
||||
---
|
||||
|
||||
# Skill Feedback
|
||||
|
||||
Follow these steps exactly:
|
||||
|
||||
1. **Ask permission**: Ask the user if they'd like you to submit feedback to the skill maintainers. Make it clear this is about the skill (the agent instructions), not about Langfuse the product. If they decline, move on.
|
||||
2. **Draft feedback**: Write the feedback using the form structure below. Present the draft to the user and ask if they'd like to change anything before submitting.
|
||||
3. **Submit**: Once approved, submit via `gh` CLI as described below. Share the resulting discussion URL with the user.
|
||||
|
||||
## Feedback Form Structure
|
||||
|
||||
Draft the feedback using these two fields:
|
||||
|
||||
**Describe your idea or feedback** (required)
|
||||
A clear description of what went wrong or what could be improved. Include:
|
||||
- What the user was trying to do
|
||||
- What the skill did vs what was expected
|
||||
- Any specific instructions that were incorrect or missing
|
||||
|
||||
**What would the ideal outcome look like?** (optional)
|
||||
What the correct behavior or guidance should be.
|
||||
|
||||
Format the body as markdown with the two field labels as headings.
|
||||
|
||||
## Submitting
|
||||
|
||||
Create a GitHub Discussion on the `langfuse/skills` repository using the GraphQL API:
|
||||
|
||||
```bash
|
||||
gh api graphql -f query='
|
||||
mutation($repoId: ID!, $categoryId: ID!, $title: String!, $body: String!) {
|
||||
createDiscussion(input: {repositoryId: $repoId, categoryId: $categoryId, title: $title, body: $body}) {
|
||||
discussion { url }
|
||||
}
|
||||
}' \
|
||||
-f repoId="$(gh api graphql -f query='{ repository(owner: "langfuse", name: "skills") { id } }' --jq '.data.repository.id')" \
|
||||
-f categoryId="$(gh api graphql -f query='{ repository(owner: "langfuse", name: "skills") { discussionCategories(first: 10) { nodes { id name } } } }' --jq '.data.repository.discussionCategories.nodes[] | select(.name == "Ideas & Improvements") | .id')" \
|
||||
-f title="<concise title>" \
|
||||
-f body="<formatted feedback>"
|
||||
```
|
||||
|
||||
If the `gh` CLI is not authenticated or the request fails, give the user this link to create the discussion manually:
|
||||
|
||||
```
|
||||
https://github.com/langfuse/skills/discussions/new?category=ideas-improvements
|
||||
```
|
||||
|
||||
After submission, share the discussion URL with the user.
|
||||
@@ -0,0 +1,88 @@
|
||||
---
|
||||
name: langfuse-user-feedback
|
||||
description: Wires up user feedback (thumbs up/down, ratings, comments) from an application's frontend to Langfuse scores. Use when user wants to capture end-user feedback, add ratings to traces, or connect user complaints to Langfuse.
|
||||
---
|
||||
|
||||
# User Feedback
|
||||
|
||||
Tracing must already be set up — feedback is stored as scores on traces.
|
||||
|
||||
Docs: https://langfuse.com/docs/observability/features/user-feedback
|
||||
|
||||
## Workflow
|
||||
|
||||
### 1. Determine What Feedback to Capture
|
||||
|
||||
If the user has asked for something specific, go with that. Otherwise, look at the application and **present a few UX options** for how feedback could work, then ask the user which they prefer before implementing.
|
||||
|
||||
Common UX patterns to suggest:
|
||||
|
||||
| UX Pattern | Best for | How it works |
|
||||
|------------|----------|--------------|
|
||||
| Thumbs up/down | Chat apps, Q&A | Simple binary buttons next to each response |
|
||||
| Star rating (1–5) | Content generation, summaries | Star row or dropdown after each output |
|
||||
| "Was this helpful?" banner | Search, documentation assistants | Single yes/no prompt at the bottom of a response |
|
||||
| Regenerate / copy tracking | Any app with these actions | Implicit — log when users retry (negative signal) or copy output (positive signal) |
|
||||
| Free-text comment | Complex outputs, internal tools | Optional text field alongside a rating |
|
||||
| Report button | Any user-facing app | Flag icon to report bad/harmful responses |
|
||||
|
||||
This table is not exhaustive — if the application suggests a different feedback pattern that fits better, propose that instead. Present 2–3 options that match the application's use case and ask the user which approach they'd like. This decision shapes everything downstream (score names, data types, frontend components), so it's important to align early.
|
||||
|
||||
Feedback can be **explicit** (user rates via thumbs, stars, etc.) or **implicit** (derived from behavior like copying output, retrying, or escalating to support). Both are stored as scores. Explicit feedback requires the trace ID to reach the frontend; implicit feedback is logged server-side where the event already happens.
|
||||
|
||||
### 2. Choose Score Names
|
||||
|
||||
Name reflects the signal source, not what you hope it measures (e.g., `user-thumbs` not `response-quality` — a thumbs down doesn't tell you *what* was wrong). Avoid generic names like `feedback` or `score`.
|
||||
|
||||
Rules:
|
||||
- Lowercase with hyphens
|
||||
- One consistent name per feedback type across the entire app
|
||||
- If capturing multiple signals, each gets its own distinct name
|
||||
|
||||
### 3. Implement Score Creation
|
||||
|
||||
**For implicit feedback (server-side):** Use `langfuse.create_score()` / `langfuse.score.create()` wherever the event is already handled in application code. Fetch SDK docs for current API: https://langfuse.com/docs/evaluation/evaluation-methods/scores-via-sdk
|
||||
|
||||
**For explicit feedback (frontend):** Use `LangfuseWeb` in the browser. It uses the public key only — no secret key exposed.
|
||||
|
||||
```typescript
|
||||
import { LangfuseWeb } from "langfuse";
|
||||
|
||||
const langfuse = new LangfuseWeb({
|
||||
publicKey: process.env.NEXT_PUBLIC_LANGFUSE_PUBLIC_KEY!,
|
||||
baseUrl: process.env.NEXT_PUBLIC_LANGFUSE_HOST,
|
||||
});
|
||||
|
||||
langfuse.score({
|
||||
traceId,
|
||||
name: "user-thumbs",
|
||||
value: 1, // 1 = positive, 0 = negative
|
||||
dataType: "BOOLEAN",
|
||||
comment: optionalUserComment,
|
||||
});
|
||||
```
|
||||
|
||||
The trace ID must be available in the frontend for this to work. For Vercel AI SDK, the non-obvious pattern is using `generateMessageId`:
|
||||
|
||||
```typescript
|
||||
import { getActiveTraceId } from "@langfuse/tracing";
|
||||
|
||||
// Inside route handler wrapped with observe()
|
||||
return result.toUIMessageStreamResponse({
|
||||
generateMessageId: () => getActiveTraceId() || crypto.randomUUID(),
|
||||
});
|
||||
```
|
||||
|
||||
### 4. Verify
|
||||
|
||||
Trigger a feedback action and check the trace's Scores tab in Langfuse. Confirm the score name, value, and data type are correct.
|
||||
|
||||
Point users to what they can do with feedback data: filter traces by low scores, use score analytics for trends, build annotation queues for team review.
|
||||
|
||||
## Common Mistakes
|
||||
|
||||
| Mistake | Problem | Fix |
|
||||
|---------|---------|-----|
|
||||
| Secret key in frontend code | Security risk | Use `LangfuseWeb` with public key only |
|
||||
| Missing `dataType` on boolean scores | Value `1` inferred as `NUMERIC` | Always pass `dataType: "BOOLEAN"` explicitly |
|
||||
| Inconsistent score names across the app | Can't aggregate or filter reliably | Pick one name per feedback type, use it everywhere |
|
||||
Reference in New Issue
Block a user