Skip to content

Impove performance when retrieving time series for set of epochs #308

@ehennestad

Description

@ehennestad

For the NDI-Marder project, profiling the element method plotEpochs uncovered some performance issue.

Profiling was done on session '412691d3730ea2a0_40cb949cfbed9eba', element: ntrode/dgn-1

Total time before making changes: 21.1 seconds

  1. Repeated calls to ndi.file.navigator/getepochfiles_number
    Todo:

    • Cache the result in a map where epoch number is the key and the resulting file path(s) is the value.
  2. Repeated calls the the database to check for ingested documents with the same query parameters.

    epoch_query = ndi.query('','isa','epochfiles_ingested') & ...
    ndi.query('','depends_on','filenavigator_id',ndi_filenavigator_obj.id()) & ...
    ndi.query('base.session_id','exact_string',ndi_filenavigator_obj.session.id());
    d_ingested = ndi_filenavigator_obj.session.database_search(epoch_query);

    This is called once per each epoch (107), resulting in total call time of ~3 seconds to mksqlite.
    @stevevanhooser What about caching the result in a temporary cache with a timeout of e.g 1 minute? Other ideas?

  3. Repeated calls to abfloaf
    There are 107 epochs, but 555 calls to read abc_header, totalling ~2 seconds.
    Todo:

    • Can this be improved?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions