Orgfetcher

Introduction

This repo contains three classes that help implement a generic interface for fetching data from an external source and using it to update an Org file. It relies on orgmunge to do the work of parsing and modifying Org files and can leverage all the power of Python to fetch external data.

The classes are Fetcher, Transformer, and Updater. Fetcher is responsible for getting data from an external source:

class Fetcher:
    """Fetches data from an external source"""
    data = None
    data_ready = False
    def __init__(self, **kwargs):
        for arg, val in kwargs.items():
            setattr(self, arg, val)
    def fetch(self):
        "Override this"
        self.data_ready = True

When you subclass Fetcher, you need to override its fetch method to do whatever you need to get the data.

This fetched data is then passed to Transformer to transform it into orgmunge Heading objects that can be used to update an existing Org file:

class Transformer:
    """Transforms fetched data into orgmunge Heading objects"""
    data = None
    data_ready = False
    todos = Org.get_todos()
    def __init__(self, fetcher: Fetcher, poll_interval: float = 5.0, **kwargs):
        self.poll_interval = poll_interval
        for arg, val in kwargs.items():
            setattr(self, arg, val)
        self.fetcher = fetcher
    def get_data(self):
        "Override this"
        self.fetcher.fetch()
        while (not self.fetcher.data_ready):
            sleep(self.poll_interval)
        self.data = self.fetcher.data
        self.transformed_data = None
    def transform(self, item: Any) -> Heading:
        "Override this. Take a data item and return a heading"
        pass

You need to override the get_data and transform methods. get_data takes the data from the fetcher, and uses the transform method on each item as appropriate to a Heading object.

The part of updating an Org file falls to Updater: it takes the Heading objects and the path to an existing Org file and figures out how to update it, whether by appending the new Heading objects or merging them with existing ones in the file:

class Updater:
    """Updates the given file using the data from the transfomer"""
    update_done = False
    def __init__(self, transformer: Transformer, org_file_path: str, **kwargs):
        self.transformer = transformer
        self.fetcher = transformer.fetcher
        self.poll_interval = transformer.poll_interval
        self.org_file_path = org_file_path
        self.todos = self.transformer.todos
        for arg, val in kwargs.items():
            setattr(self, arg, val)
    def _get_data(self):
        self.fetcher.fetch()
        self.transformer.get_data()
        while(not self.transformer.data_ready):
            sleep(self.poll_interval)
    def update_headings(self):
        "Override this"
        if hasattr(self, 'todos'):
            self.org_file = Org(self.org_file_path, todos=self.todos)
        else:
            self.org_file = Org(self.org_file_path)
    def update(self):
        self._get_data()
        self.update_headings()
        self.org_file.write(self.org_file_path)

Most likely, you only need to override the update_headings method. You can take the data from the transformer and decide how to merge it with headings in the existing Org file.

Example

The repo also contains an example file (github_issues.py) showing how to inherit from these classes to create a system that fetches issues from all your github repos and updates an Org file: any new issues that were opened get added under a top-level heading named after that repo; any new issues that were closed, are marked as done in the file and their closed time is set to the time the corresponding issue was closed on github. In addition, the system also adds the agenda-group property to each repo’s heading and sets it to the dominant language in that repo. This helps create a nice grouping of issues when you query them with org-ql.

License