Skip to content

Add compatibility with arrow to deal with larger-than-memory data #48

@rafapereirabr

Description

@rafapereirabr

Problem:
Oftentimes, travel cost matrices can get really big in large study areas. This creates a challenge of dealing with larger-than-memory data. The current version of {accessibility} cannot cope with this problem. Although the package is super efficient because it is built on top od {data.table}, the package still requires users to load the travel_matrix and land_use_data input to RAM memory, which can be a constraint for these larger-than-memory cases.

Proposed solution:
One solution to this would be to make the {accessibility} compatible with {arrow} (link). There are different ways to do this. The alternative I would suggest would use the following workfkow:

  • If the user passes travel_matrix and land_use_data inputs as data.tables or data.frames, then the package follows the current natural flow.
  • If the user passes a string path pointing to the location of .parquet or .csv files that contain the travel_matrix and land_use_data inputs, then the package follows a different flow to calculate the indicador using a {dplyr} + {arrow} syntax

Implications:

  • For users, this only means more flexibility to work with larger-than-memory data and greater computation speed.
  • For developers, this means we would need to make a few changes to check inputs, but most of the work would be translating the accessibility functions into a {dplyr} syntax, which is generally super simple.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions