Speed up reads -- parallel parsing!

[oyeb]

I'm not sure what's a recent pythonic way to do multi-threading? Are Futures and Promises now commonplace?

[adhuliya]

AFAIK, GIL is still in place. A plain old threading model should suffice. I have recently used it in one of my scripts. Something like below (from the script) should work when spawned on files,

while True:
  thread = threading.Thread(target=sendRequest)
  thread.start()
  # limit the number of threads
  while threading.activeCount() >= MAX_THREADS:
    now = datetime.now()
    print(f"{now}: MAX_THREADS = {MAX_THREADS}")
    time.sleep(5)
  time.sleep(SLEEP_TIME)

Moreover, if we are collecting everything in one data structure then
its access has to be synchronized.