Skip to content

coherence_strategy.md

Latest commit

 

History

History
71 lines (53 loc) · 5.5 KB

coherence_strategy.md

File metadata and controls

71 lines (53 loc) · 5.5 KB

Knowledge Graph Coherence Maintenance Strategy

1. Introduction

Maintaining a coherent and up-to-date knowledge graph is crucial for its accuracy, reliability, and overall usefulness. As web content changes and new information is discovered, the graph must adapt to reflect these changes accurately. This document outlines the strategy for achieving and maintaining this coherence.

2. Core Principles

  • Data Freshness: Ensuring that the information stored for web resources reflects their current state on the live web.
  • Link Validity: Verifying that links between resources remain active and relevant, removing or flagging dead or broken links.
  • Information Consistency: Maintaining logical consistency within the graph, such as ensuring that resources have necessary attributes and that linked resources are topically related.

3. Key Processes & Algorithms

Resource Re-crawling

  • Decision for Re-crawl: Resources will be prioritized for re-crawling based on factors like:
    • Age: Time since the last_crawled_at timestamp.
    • Importance: (Future) A calculated score based on centrality, number of incoming links, or explicit user interest.
    • Change Frequency: (Future) Historical data on how often a resource's content changes.
  • Role of content_hash: A hash (e.g., MD5 or SHA256) of the raw fetched content will be stored. Upon re-crawl, if the new content_hash matches the stored one, further processing (parsing, content extraction) can be skipped, saving resources.
  • Content Change Handling: If the content_hash differs:
    • The new content is processed by ContentProcessor.
    • The resource's content, processed_text, metadata, and last_crawled_at fields are updated in GraphDBInterface.
    • The resource and its immediate neighborhood (connected links and resources) are flagged for re-analysis by ConsistencyAnalyzer.

Link Management

  • Validating Existing Links: Periodically, or during re-crawl of a source page:
    • Target URLs of outgoing links can be checked for HTTP status codes (e.g., 200 OK, 404 Not Found, redirects).
    • Links leading to 404s or permanent redirects might be flagged, have their status updated, or eventually removed.
  • Handling Disappeared Links: If a previously recorded link is no longer present on a source page after it's re-crawled:
    • The link's status in GraphDBInterface can be updated (e.g., "inactive") or the link can be removed.
    • This might trigger consistency checks, especially if the link was considered important.

New Content Integration

  • Flow:
    1. LinkDiscoverer identifies potential new URLs (from existing links pointing to uncrawled targets or from search suggestions).
    2. WebCrawler fetches content for these URLs.
    3. ContentProcessor extracts text and relevant information from the fetched content.
    4. GraphDBInterface stores the new resource and any new links found on that page.
  • Triggering Consistency Checks:
    • Newly added resources are immediately checked for completeness by ConsistencyAnalyzer.
    • New links, and the resources they connect, are checked for topical coherence.

Periodic Audits

  • Full/Subset Analysis: ConsistencyAnalyzer will be run periodically (e.g., nightly, weekly) on:
    • The entire graph to find global inconsistencies.
    • Subsets of the graph that have recently changed or are deemed critical.
  • Refreshing Discovery Pool: LinkDiscoverer will be run periodically to:
    • Identify new uncrawled links that have appeared in existing content.
    • Generate new search queries based on the current state of the graph's topics, feeding new entry points for crawling.

4. Module Interactions Summary

  • WebCrawler: Fetches new and updated content, providing the raw data for coherence checks. Its efficiency (e.g., respecting robots.txt, handling content_hash) is key.
  • ContentProcessor: Extracts clean text and metadata. Changes in its output directly impact resource representation and thus consistency.
  • GraphDBInterface: The central store. It must accurately reflect updates, deletions, and status changes resulting from coherence processes. Placeholder creation for linked but uncrawled URLs is a key feature for maintaining relational integrity.
  • ConsistencyAnalyzer: Actively identifies issues (incomplete data, incoherent links) by querying GraphDBInterface. Its findings trigger corrective actions or further investigation.
  • LinkDiscoverer: Drives graph expansion. Its suggestions for new URLs to crawl are essential for keeping the graph current and comprehensive. It relies on GraphDBInterface to know what's already crawled.

5. Future Enhancements (Briefly)

  • Automated Conflict Resolution: Developing rules or models to automatically resolve certain types of inconsistencies or conflicting information found in different resources.
  • User Feedback Loops: Incorporating mechanisms for users to report errors, suggest changes, or validate information, feeding back into the coherence processes.
  • Advanced Anomaly Detection: Using machine learning to detect unusual patterns or outliers in the graph data that might indicate subtle inconsistencies or emerging topics.
  • Probabilistic Link Strength: Assigning confidence scores to links based on coherence and source reliability.

This strategy aims to create a resilient and adaptive knowledge graph that remains a valuable asset over time.