[ENHANCEMENT] Add file context tracking to skip redundant re-reads (reduces token usage)

[hannesrudolph]

Type

Enhancement

Problem / Value

When using read_file operations, the system re-reads files even when they haven't been modified since the last read and the content is still present in the conversation context. This results in unnecessary token usage and reduced efficiency.

Context

Affects all users performing file reads during coding sessions. When a file is unchanged and its content is still in the conversation context (not condensed or truncated), there is no need to re-read the entire file. The system should detect this situation and return a short response instead, reducing token usage and improving overall efficiency.

[roomote]

I've analyzed this enhancement request and explored the codebase. Here's my proposed implementation plan:

Analysis

The codebase already has infrastructure that can be leveraged:

• FileContextTracker (src/core/context-tracking/FileContextTracker.ts) - tracks file operations with timestamps (roo_read_date, user_edit_date, roo_edit_date)
• File watchers - detect external modifications to tracked files
• Context condensation (src/core/condense/index.ts) - handles context summarization with condenseParent tracking

What's missing is the ability to:

1. Detect unchanged files that are still in the conversation context
2. Skip re-reading those files to save tokens

Proposed Implementation Plan

1. Extend FileMetadataEntry to include content hash

File: src/core/context-tracking/FileContextTrackerTypes.ts

Add a content_hash field to track file content at read time:

export const fileMetadataEntrySchema = z.object({
  // ... existing fields
  content_hash: z.string().nullable(), // MD5/SHA hash of file content when last read
})

2. Modify FileContextTracker to store content hash

File: src/core/context-tracking/FileContextTracker.ts

• Update trackFileContext() to accept an optional content hash parameter
• When a file is read via read_tool, compute and store the hash of its content
• Add method isFileUnchangedInContext(filePath) that checks:
  • File has been read (roo_read_date exists)
  • No user edits after the read (user_edit_date <= roo_read_date or null)
  • File content hash matches current file

3. Track context validity for file reads

File: src/core/context-tracking/FileContextTracker.ts

Add a method to invalidate file context when condensation occurs:

• When context is condensed, mark file entries' context as potentially stale
• Add context_valid_from_ts field to track when file content entered the active context
• Compare against condensation timestamps to determine if content is still in context

4. Modify ReadFileTool to skip redundant re-reads

File: src/core/tools/ReadFileTool.ts

Before reading a file, check if it's already in active context with unchanged content:

// In execute() method, before file reading logic:
const isInContext = await task.fileContextTracker.isFileUnchangedInContext(relPath)
if (isInContext) {
  // Return short response instead of full file content
  updateFileResult(relPath, {
    xmlContent: `<file><path>${relPath}</path><notice>File unchanged since last read. Content is already in context.</notice></file>`,
    nativeContent: `File: ${relPath}\nNote: File unchanged since last read. Content is already in context.`,
  })
  continue
}

5. Integrate with context condensation

File: src/core/condense/index.ts

When context is condensed:

• Notify FileContextTracker to mark affected file entries
• Files whose read timestamp is older than the condensation point should be marked for re-read

Benefits

• Reduced token usage - unchanged files return ~20 tokens instead of potentially thousands
• Maintained correctness - files are re-read when:
  • Modified externally (detected by file watchers)
  • Modified by Roo
  • Context is condensed/truncated past the read operation
• Integrates with existing infrastructure - leverages FileContextTracker and file watchers

Testing Plan

1. Unit tests for new FileContextTracker methods
2. Integration tests for ReadFileTool skip logic
3. End-to-end tests verifying:
   • Unchanged file returns short response
   • Modified file returns full content
   • File is re-read after context condensation

Please mention @roomote in a comment saying that you approve this plan, and I'll proceed with the implementation.

[dreness]

@roomote I'd be interested to see if your proposed approach helps; please proceed with a PR if you're willing :)

[roomote]

Thanks for the go-ahead @dreness! I'll proceed with implementing the proposed plan and will open a PR shortly to share my progress. Looking forward to feedback on the approach.

[roomote]

Opened a PR with my attempt at addressing this: #10807

The implementation adds content hashing to the FileContextTracker to detect unchanged files and skip redundant re-reads in ReadFileTool. Feedback welcome!