Skip to content

parallelize file creation #6

@pilosoposerio

Description

@pilosoposerio

this is for another enhancement

cli runs too long, maybe multiprocessing module can be used to improve the performance of this block

for count, features in enumerate(gj.stream(batch=args.geometry_count)):
try:
new_filename: Path = gen_filename(
gj.geojson, count, width=args.suffix_length, parent=args.output
)
except TypeError as e:
logger.error(f"Could not generate a unique suffix.", exc_info=e)
try:
if not args.dry_run:
if not new_filename.parent.exists():
logger.debug(f"creating output directory {args.output}")
new_filename.parent.mkdir(parents=True, exist_ok=True)
with new_filename.open("w") as fp:
json.dump(features, fp)
logger.debug(
f"successfully saved {len(features['features'])} features to {new_filename}"
)
except IOError as e:
logger.error(f"Could not write features to {new_filename}", exc_info=e)
# account for 0 based index of enumerate that is required for `pad` method.
if args.limit is not None:
if count >= args.limit - 1:
break

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions