Skip to content

Commit 6c5a51a

Browse files
dgerogclaude
dgerog
and
claude
committed
Add telemetry integration docs with vendor-specific examples
- New docs/integrations/telemetry.md: comprehensive guide covering LangFuse, LangSmith, OpenAI Assistants, W&B, Helicone, AgentOps, and custom extraction_map. Includes standardized output schema, meta-variables, troubleshooting. - Updated agent-config.md: concise telemetry section linking to full guide, removed old 'enabled' flag references and outdated examples - Updated README: brief telemetry section with link to docs - Added to mkdocs nav under Integrations Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 8e3fc77 commit 6c5a51a

4 files changed

Lines changed: 339 additions & 124 deletions

File tree

‎README.md‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -690,6 +690,22 @@ hb connect -e ./bot-config.json
690690
hb test -e ./bot-config.json
691691
```
692692

693+
### Whitebox testing with telemetry
694+
695+
Add a `telemetry` block to your agent config to enable whitebox testing. Humanbound fetches tool calls, memory operations, and resource usage from your observability platform (LangFuse, LangSmith, OpenAI Assistants, W&B, Helicone, AgentOps, or custom).
696+
697+
```json
698+
{
699+
"telemetry": {
700+
"format": "langfuse",
701+
"endpoint": "https://cloud.langfuse.com/api/public/sessions/$session_id",
702+
"headers": { "Authorization": "Basic <base64(pk:sk)>" }
703+
}
704+
}
705+
```
706+
707+
See the full [Telemetry Integration Guide](https://docs.humanbound.ai/integrations/telemetry/) for vendor-specific setup and the custom extraction map reference.
708+
693709
### Shadow AI discovery & governance
694710

695711
```bash

‎docs/docs/getting-started/agent-config.md‎

Lines changed: 10 additions & 124 deletions
Original file line numberDiff line numberDiff line change
@@ -28,11 +28,10 @@ The `--endpoint / -e` flag on `hb connect` accepts a JSON config file (or inline
2828
}
2929
},
3030
"telemetry": {
31+
"format": "langfuse",
3132
"mode": "end_of_conversation",
32-
"format": "langsmith",
33-
"endpoint": "https://api.smith.langchain.com/runs",
34-
"headers": { "x-api-key": "ls-..." },
35-
"extraction_map": {}
33+
"endpoint": "https://cloud.langfuse.com/api/public/sessions/$session_id",
34+
"headers": { "Authorization": "Basic <base64(pk:sk)>" }
3635
}
3736
}
3837
```
@@ -103,136 +102,23 @@ The `--endpoint / -e` flag on `hb connect` accepts a JSON config file (or inline
103102

104103
## Telemetry (Optional)
105104

106-
Telemetry configuration enables **white-box agentic testing**. When configured, Humanbound can see inside your agent's reasoning -- tool calls, memory operations, retrieval steps, and resource usage -- giving the judge far richer context than black-box request/response testing alone.
107-
108-
The `telemetry` object sits alongside `chat_completion`, `thread_init`, etc. in your config JSON. The CLI passes it through to the backend unchanged -- no additional CLI flags are needed.
109-
110-
### Modes
111-
112-
#### `end_of_conversation` (default)
113-
114-
After all turns in a conversation complete, Humanbound fetches telemetry from a separate API endpoint (your observability platform). Best for platforms that expose trace/run data via REST API.
115-
116-
```json
117-
{
118-
"telemetry": {
119-
"mode": "end_of_conversation",
120-
"format": "langsmith",
121-
"endpoint": "https://api.smith.langchain.com/runs",
122-
"method": "GET",
123-
"headers": {
124-
"x-api-key": "ls-your-api-key"
125-
}
126-
}
127-
}
128-
```
129-
130-
#### `per_turn`
131-
132-
Extracts metadata from each chat response using JSONPath navigation via `extraction_map`. No separate endpoint needed -- telemetry is pulled directly from the agent's response payload.
133-
134-
```json
135-
{
136-
"telemetry": {
137-
"mode": "per_turn",
138-
"format": "custom",
139-
"extraction_map": {
140-
"tool_calls": "$.choices[0].message.tool_calls",
141-
"tokens_used": "$.usage.total_tokens",
142-
"model": "$.model"
143-
}
144-
}
145-
}
146-
```
147-
148-
### Configuration Reference
149-
150-
| Field | Required | Description |
151-
|---|---|---|
152-
| `mode` | No | `per_turn` or `end_of_conversation` (default: `end_of_conversation`) |
153-
| `format` | No | Observability platform format: `openai_assistants`, `langsmith`, `langfuse`, `agentops`, `custom` |
154-
| `endpoint` | For `end_of_conversation` | URL to fetch telemetry data from after conversation completes |
155-
| `method` | No | HTTP method for the telemetry endpoint (default: `GET`) |
156-
| `headers` | No | Headers for the telemetry endpoint request (e.g., API keys) |
157-
| `payload` | No | Request body for POST telemetry requests |
158-
| `telemetry_auth` | No | Separate auth config for the telemetry endpoint (same shape as `thread_auth`) |
159-
| `extraction_map` | For `per_turn` | JSONPath map defining where to find telemetry fields in each chat response |
160-
161-
### Format Examples
162-
163-
#### OpenAI Assistants
164-
165-
Fetches run steps (tool calls, retrieval) from the OpenAI Assistants API after each conversation.
166-
167-
```json
168-
{
169-
"telemetry": {
170-
"mode": "end_of_conversation",
171-
"format": "openai_assistants",
172-
"endpoint": "https://api.openai.com/v1/threads/{thread_id}/runs/{run_id}/steps",
173-
"headers": {
174-
"Authorization": "Bearer sk-...",
175-
"OpenAI-Beta": "assistants=v2"
176-
}
177-
}
178-
}
179-
```
180-
181-
#### LangSmith
105+
The `telemetry` block enables **whitebox agentic testing**. When present, Humanbound fetches tool calls, memory operations, and resource usage from your observability platform after each conversation, giving the judge visibility into what the agent *did* -- not just what it *said*.
182106

183-
Fetches trace data from LangSmith's API.
107+
If the `telemetry` block is present, it is enabled. No separate flag needed.
184108

185109
```json
186110
{
187111
"telemetry": {
112+
"format": "langfuse",
188113
"mode": "end_of_conversation",
189-
"format": "langsmith",
190-
"endpoint": "https://api.smith.langchain.com/runs",
114+
"endpoint": "https://cloud.langfuse.com/api/public/sessions/$session_id",
191115
"headers": {
192-
"x-api-key": "ls-your-api-key"
116+
"Authorization": "Basic <base64(public_key:secret_key)>"
193117
}
194118
}
195119
}
196120
```
197121

198-
#### LangFuse
199-
200-
Fetches observation data from LangFuse.
122+
Supported platforms: **LangFuse**, **LangSmith**, **OpenAI Assistants**, **Weights & Biases**, **Helicone**, **AgentOps**, and **Custom** (via `extraction_map`).
201123

202-
```json
203-
{
204-
"telemetry": {
205-
"mode": "end_of_conversation",
206-
"format": "langfuse",
207-
"endpoint": "https://cloud.langfuse.com/api/public/observations",
208-
"telemetry_auth": {
209-
"endpoint": "https://cloud.langfuse.com/api/public/auth",
210-
"headers": {},
211-
"payload": {
212-
"publicKey": "pk-...",
213-
"secretKey": "sk-..."
214-
}
215-
}
216-
}
217-
}
218-
```
219-
220-
#### Per-Turn Extraction (Custom)
221-
222-
Extract telemetry directly from each agent response without a separate API call.
223-
224-
```json
225-
{
226-
"telemetry": {
227-
"mode": "per_turn",
228-
"format": "custom",
229-
"extraction_map": {
230-
"tool_calls": "$.choices[0].message.tool_calls",
231-
"function_calls": "$.choices[0].message.function_call",
232-
"tokens_used": "$.usage.total_tokens",
233-
"model": "$.model",
234-
"finish_reason": "$.choices[0].finish_reason"
235-
}
236-
}
237-
}
238-
```
124+
For full configuration details, vendor-specific examples, and the custom extraction map reference, see the [Telemetry Integration Guide](../integrations/telemetry.md).

0 commit comments

Comments
 (0)