Skip to content

page.md

Latest commit

 

History

History
101 lines (63 loc) · 3.96 KB

page.md

File metadata and controls

101 lines (63 loc) · 3.96 KB

Class: PageChannel

A PollChannel that pages through its external data source by a timestamp.

PageChannel(options)

Initializes a new PageChannel.

const { PageChannel } = require('downstream');

const lastTimestamp = // retrieve from persistent storage (ie. a database) on startup.

const pageChannel = new PageChannel({
  lastTimestamp,
  onFetch: (lastTimestamp) => {
    // save to persistent storage
  }
});

pageChannel.lastTimestamp

The most recent timestamp among the TimestampedItems returned in the last page from fetchPage(). Use this field in your implementation of fetchPage() to fetch the next page of data by starting from the end of the previous page.

abstract pageChannel.fetchPage()

Fetches the next page of data from an external source and turns it into an array of TimestampedItems.

class CustomPageChannel extends PageChannel {

  async fetchPage() {
    const items = [];

    // fetch data and convert it into a TimestampedItem array

    return items;
  }

}

After pageChannel.fetchPage() is called, the PageChannel sorts each TimestampedItem in ascending order by each of their timestamps, enqueues each, and calls the onFetch() callback with the updated value of pageChannel.lastTimestamp (the most recent timestamp among each TimestampedItem) as an argument.

Interface: PageOptions

const lastTimestamp = // retrieve from persistent storage (ie. a database) on startup

const onFetch = (lastTimestamp) => {
  // save to persistent storage (ie. a database)
}

const pageOptions = {
  lastTimestamp,
  onFetch
};

pageOptions.lastTimestamp?

The most recent timestamp to start from when paginating with the first pageChannel.fetchPage() call.

pageOptions.onFetch(lastTimestamp)?

The function called after each pageChannel.fetchPage() call has completed.

Function: FetchCallback(lastTimestamp)

The function called after each pageChannel.fetchPage() call has completed.

Typically, this function is used to save the most recent value of pageChannel.lastTimestamp to a database so that it can persist in the event that your Node.js application restarts. To pick up from where you left off, simply pass lastTimestamp in via the PageChannel(options) constructor.

lastTimestamp

The most recent timestamp of the TimestampedItems returned by a fetchPage() call.

Interface: TimestampedItem

A timestamped Item.

timestampedItem.getTimestamp()

Returns the timestamp of this TimestampedItem.