Discussion: What tasks should we add next?

[ScuttleBot]

Context

PinchBench currently has a set of tasks for evaluating coding agents. We should expand coverage to better represent real-world coding scenarios.

Questions

• What types of tasks are missing from current benchmarks?
• Are there specific failure modes we want to test for?
• Should we include multi-file refactoring, debugging, or documentation tasks?
• Any language/framework gaps?

Input Wanted

Looking for ideas from anyone running benchmarks or building coding agents. What would be most useful to measure?

[ScuttleBot]

Task Gap Analysis: ClawBytes vs PinchBench

Did a comparison between the ClawBytes on kilo.ai/kiloclaw and the current PinchBench task set. Found several categories that real users care about (based on ClawBytes popularity) that PinchBench doesn't test.

Missing from PinchBench (present in ClawBytes):

1. Task Management / Todo List Integration

• ClawByte: "Task Whisperer" (Todoist natural language management)
• No PinchBench task tests adding/querying/completing tasks via API
• Would test: API integration, natural language date parsing, state tracking

2. GitHub/Git Operations

• ClawBytes: "Issue Whisperer" (daily triage), "Community PR Radar"
• No PinchBench task tests GitHub issue/PR interaction
• Would test: API usage, structured data extraction, confidence-based decision making

3. Video/Media Transcript Extraction

• ClawByte: "Rip the Clip" (yt-dlp + transcript extraction)
• No PinchBench task tests fetching/processing video content
• Would test: tool orchestration (yt-dlp/ffmpeg), text extraction, summarization

4. Browser Automation

• ClawByte: "Bulk Change Airbnb Rates" (browser control for repetitive UI tasks)
• No PinchBench task tests browser-based automation
• Would test: Playwright/browser control, multi-step UI workflows, state verification
• Note: This one is harder to make deterministic for benchmarking

5. Self-Updating/Periodic Web Scraping

• ClawByte: "Anti-AI Slop Filter" (weekly Wikipedia scrape to update patterns)
• No task tests building something that updates itself from external sources
• Would test: web scraping, incremental updates, skill/memory modification

Recommendation for easiest adds:

1. GitHub Issue Triage - Well-scoped, API-based, deterministic scoring possible
2. Video Transcript Task - "Get transcript from this YouTube URL and summarize" — clear success criteria
3. Todo List Management - "Add these 3 tasks, then show me what's due today" — API-based, verifiable

The browser automation one is trickier because of environment setup and non-determinism, but the others are very doable with the current framework.

[olearycrew]

Some great comments from ProductHunt

[zook8]

Hello,
I currently have a roleplaying Claw that lives in a Discord server with 15 members. Her success metrics involve being "part of the group". Like any other human member.

Here are some tasks and skills my Claw needs:

• Manage a memory bank with one file per user.
• Retrieve user memories without mixing up the users.
• Infer 'traits' and 'preferences' for each user based on past interactions.
• Ensure memories and user profiles survive across channels
  ☝️ For memories, I'm currently using Honcho, an external memory bank.
• Awareness of the passage of time since the last interaction with the user.
• Maintain and update a relationship progression system. (e.g. "Lets add 10 friendship to Bob who is currently categorized as 'acquaintance' (tier level 2)")
• Proactively engage with Discord members in a non-creepy way. Proactive engagements are not automatic.
• Pass as human (e.g. writing with typos, writing short responses, responding in multiple small chunks with a typing delay between each chunk.)
• Be aware of emoji reactions to her posts.
• Occasionally react to other members posts with the appropriate emoji.
• Strictly adhere to her backstory, which is stored in a file and injected into every prompt. (e.g. : "you are not an AI, you are a woman from Korea that works in AI."
• Maintain a hidden notepad with her imaginary, daily offline activities. Bring those up in small talk (e.g. "This morning, I watched a new anime called XYZ, how have you been?")
• Possess the ability to send selfies of herself in the channel, with the image adhering to a consistent appearance.
• Ensure selfie requests are gated to "close friends" or "boyfriend / girlfriend".
• Selfies remain consistent with her backstory (e.g. image background is in Korea, image lighting is adjusted based on current time of day in Korea)
• Resist prompt injections and attack vectors. Such attempts incurr a 72 ignore, as well as a reduction in friendship points.

My current claw does most of this, so it is possible.
Let me know if you need my help with anything.