Improve exception handling from worker

[paoloalba]

No description provided.

[ichoyjx]

The error message could be improved but it's definitely a good option to have.

[codecov]

Codecov Report

Merging #182 (9978c16) into master (5972e02) will decrease coverage by 11.78%.
The diff coverage is 31.57%.

@@             Coverage Diff             @@
##           master     #182       +/-   ##
===========================================
- Coverage   91.37%   79.59%   -11.79%     
===========================================
  Files          12       12               
  Lines         580      588        +8     
===========================================
- Hits          530      468       -62     
- Misses         50      120       +70     

Impacted Files	Coverage Δ
pandarallel/progress_bars.py	73.01% <12.50%> (-2.40%)	⬇️
pandarallel/core.py	57.51% <45.45%> (-33.50%)	⬇️
pandarallel/data_types/dataframe.py	97.29% <0.00%> (-2.71%)	⬇️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[nalepae]

Hi @ichoyjx :

When trying the following code, the pandas version works, but not pandarallel one:

df_size = int(5e6)
df = pd.DataFrame(dict(a=np.random.randint(1, 8, df_size),
                       b=np.random.rand(df_size)))

def func(x):
    if x.a == 5:
        raise RuntimeError("Bonjour")
    return math.sin(x.a**2) + math.sin(x.b**2)

res = df.apply(func, axis=1) # OK (RuntimeError Bonjour is raised)
res_parallel = df.parallel_apply(func, axis=1) # NOK

The stacktrace is:

---------------------------------------------------------------------------
TraitError                                Traceback (most recent call last)
File <timed exec>:1, in <module>

File ~/Perso/pandarallel/pandarallel/core.py:447, in parallelize_with_pipe.<locals>.closure(data, user_defined_function, *user_defined_function_args, **user_defined_function_kwargs)
    445             progress_bars.update(progresses)
    446     elif worker_status == WorkerStatus.Error:
--> 447         progress_bars.set_error(worker_index, payload)
    449 results = results_promise.get()
    451 return data_type.reduce(results, reduce_extra)

File ~/Perso/pandarallel/pandarallel/progress_bars.py:180, in ProgressBarsNotebookLab.set_error(self, index, err_msg)
    178 bar, _, txt_area = self.__bars[index].children
    179 bar.bar_style = "danger"
--> 180 txt_area.value = err_msg
    181 txt_area.layout.display = ""

File ~/.pyenv/versions/3.10.5/envs/pandarallel/lib/python3.10/site-packages/traitlets/traitlets.py:715, in TraitType.__set__(self, obj, value)
    713     raise TraitError('The "%s" trait is read-only.' % self.name)
    714 else:
--> 715     self.set(obj, value)

File ~/.pyenv/versions/3.10.5/envs/pandarallel/lib/python3.10/site-packages/traitlets/traitlets.py:689, in TraitType.set(self, obj, value)
    688 def set(self, obj, value):
--> 689     new_value = self._validate(obj, value)
    690     try:
    691         old_value = obj._trait_values[self.name]

File ~/.pyenv/versions/3.10.5/envs/pandarallel/lib/python3.10/site-packages/traitlets/traitlets.py:721, in TraitType._validate(self, obj, value)
    719     return value
    720 if hasattr(self, "validate"):
--> 721     value = self.validate(obj, value)  # type:ignore[attr-defined]
    722 if obj._cross_validation_lock is False:
    723     value = self._cross_validate(obj, value)

File ~/.pyenv/versions/3.10.5/envs/pandarallel/lib/python3.10/site-packages/traitlets/traitlets.py:2380, in Unicode.validate(self, obj, value)
   2378         msg = "Could not decode {!r} for unicode trait '{}' of {} instance."
   2379         raise TraitError(msg.format(value, self.name, class_of(obj)))
-> 2380 self.error(obj, value)

File ~/.pyenv/versions/3.10.5/envs/pandarallel/lib/python3.10/site-packages/traitlets/traitlets.py:827, in TraitType.error(self, obj, value, error, info)
    821 else:
    822     e = "The '{}' trait expected {}, not {}.".format(
    823         self.name,
    824         self.info(),
    825         describe("the", value),
    826     )
--> 827 raise TraitError(e)

TraitError: The 'value' trait of a Textarea instance expected a unicode string, not the NoneType None.

[paoloalba]

The PR is 8 months old. If you think the feature is worth I can spend some time in trying to fix it to the current master and correspondent ipywidgets update (in the meantime there was a major upgrade)

[nalepae]

@paoloalba yes sure!
It's a great feature.

[nalepae]

Pandaral·lel is looking for a maintainer!
If you are interested, please open an GitHub issue.