Skip to content

Multiprocessing breaks Spheroscope on Apple Silicon #93

@schtepf

Description

@schtepf

The multiprocessing module is known to have issues on recent MacOS versions (at least with Apple Silicon hardware). Its unconditional use in Spheroscope seems to break all functionality that involves running multiple queries.

The error message (included below) is difficult to decipher, as it complains about a missing Flask application context. If this doesn't happen on other platforms, it is probably some side effect of multiprocessing bugs on MacOS.

multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/opt/anaconda3/envs/Spheroscope/lib/python3.9/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/opt/anaconda3/envs/Spheroscope/lib/python3.9/multiprocessing/pool.py", line 51, in starmapstar
    return list(itertools.starmap(args[0], args[1]))
  File "/Users/ex47emin/Project/RANT/Software/Spheroscope-GIT/spheroscope/queries.py", line 23, in run_query
    corpus_config = read_config(cwb_id)
  File "/Users/ex47emin/Project/RANT/Software/Spheroscope-GIT/spheroscope/corpora.py", line 59, in read_config
    corpus_dir = os.path.join(current_app.instance_path, cwb_id)
  File "/opt/anaconda3/envs/Spheroscope/lib/python3.9/site-packages/werkzeug/local.py", line 311, in __get__
    obj = instance._get_current_object()
  File "/opt/anaconda3/envs/Spheroscope/lib/python3.9/site-packages/werkzeug/local.py", line 508, in _get_current_object
    raise RuntimeError(unbound_message) from None
RuntimeError: Working outside of application context.

This typically means that you attempted to use functionality that needed
the current application. To solve this, set up an application context
with app.app_context(). See the documentation for more information.
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/anaconda3/envs/Spheroscope/lib/python3.9/site-packages/flask/app.py", line 2552, in __call__
    return self.wsgi_app(environ, start_response)
  File "/opt/anaconda3/envs/Spheroscope/lib/python3.9/site-packages/flask/app.py", line 2532, in wsgi_app
    response = self.handle_exception(e)
  File "/opt/anaconda3/envs/Spheroscope/lib/python3.9/site-packages/flask/app.py", line 2529, in wsgi_app
    response = self.full_dispatch_request()
  File "/opt/anaconda3/envs/Spheroscope/lib/python3.9/site-packages/flask/app.py", line 1825, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/opt/anaconda3/envs/Spheroscope/lib/python3.9/site-packages/flask/app.py", line 1823, in full_dispatch_request
    rv = self.dispatch_request()
  File "/opt/anaconda3/envs/Spheroscope/lib/python3.9/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/Users/ex47emin/Project/RANT/Software/Spheroscope-GIT/spheroscope/auth.py", line 97, in wrapped_view
    return view(**kwargs)
  File "/Users/ex47emin/Project/RANT/Software/Spheroscope-GIT/spheroscope/patterns.py", line 187, in matches
    matches = run_queries(queries, cwb_id)
  File "/Users/ex47emin/Project/RANT/Software/Spheroscope-GIT/spheroscope/queries.py", line 101, in run_queries
    matches = pool.starmap(run_query, [(query, cwb_id) for query in queries])
  File "/opt/anaconda3/envs/Spheroscope/lib/python3.9/multiprocessing/pool.py", line 372, in starmap
    return self._map_async(func, iterable, starmapstar, chunksize).get()
  File "/opt/anaconda3/envs/Spheroscope/lib/python3.9/multiprocessing/pool.py", line 771, in get
    raise self._value
RuntimeError: Working outside of application context.

This typically means that you attempted to use functionality that needed
the current application. To solve this, set up an application context
with app.app_context(). See the documentation for more information.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions