Skip to content

Bug: Argentina preprocess doesn´t skip based on variable #205

@outerlook

Description

@outerlook

Problem

preprocess_flow.py processes ALL available dates instead of respecting LAST_PREPROCESS_SUCCESS_DATE like other Argentina flows do.

Current behavior (line 188-194):

for date in self.raw_provider.list_available_keys():
    if self.processed_provider.exists(date):
        continue
    await self.process_date(date)

Expected behavior: Filter dates by LAST_PREPROCESS_SUCCESS_DATE first, then check if already processed.

Solution

Add date filtering before the loop:

last_preprocess_date = await variables.Variable.aget(
    ArgentinaFlowVariableNames.LAST_PREPROCESS_SUCCESS_DATE,
    default=ArgentinaFlowVariableNames.DEFAULT_DATE
)
dates_to_process = [d for d in self.raw_provider.list_available_keys() if d > last_preprocess_date]

This matches the pattern already used in aggregate_products_flow.py and insert_products_flow.py.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Status

    No status

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions