Time jitter at high frequencies

[martijnende]

I'm looking at a data set acquired at 2500 Hz which was not GPS synchronised. When I try to load a time slice of data, xdas complains about the jitter in the time samples, and asks me to do a .simplify(). When I attempt to do this, I get a somewhat exotic NumPy dtype error.

Minimal example:

import xdas
# print(xdas.__version__)  # does not exist, apparently...

path = "./data/{cable}/[acquisition].hdf5"
dc = xdas.open_mfdatatree(path, engine="asn")
da = dc["bug"][0]

t_slice = slice("2025-07-29T23:53:20", "2025-07-29T23:53:30")
dist_slice = slice(5.7e3, 6.e3)
data = da.sel(distance=dist_slice, time=t_slice)

This raises:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
File ~/miniconda3/envs/abyss/lib/python3.11/site-packages/xdas/core/coordinates.py:775, in InterpCoordinate.get_indexer(self, value, method)
    774 try:
--> 775     indexer = inverse(value, self.tie_indices, self.tie_values, method)
    776 except ValueError as e:

File ~/miniconda3/envs/abyss/lib/python3.11/site-packages/xinterp/core.py:61, in inverse(f, xp, fp, method)
     33 """
     34 One-dimensional linear interpolation from values to indices.
     35 
   (...)     59     If any value of `f` is outside the `fp` range.
     60 """
---> 61 return _inverse(xp, fp, f=f, method=method)

File ~/miniconda3/envs/abyss/lib/python3.11/site-packages/xinterp/core.py:66, in wraps.<locals>.func(xp, fp, x, f, **kwargs)
     65 def func(xp, fp, *, x=None, f=None, **kwargs):
---> 66     xp, fp, x, f, isscalar = check(xp, fp, x, f)
     67     if np.issubdtype(fp.dtype, np.integer) or np.issubdtype(
     68         fp.dtype, np.datetime64
     69     ):

File ~/miniconda3/envs/abyss/lib/python3.11/site-packages/xinterp/core.py:143, in check(xp, fp, x, f)
    142     if not np.all(fp[1:] > fp[:-1]):
--> 143         raise ValueError("fp must be strictly increasing")
    144 return xp, fp, x, f, isscalar

ValueError: fp must be strictly increasing

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
Cell In[14], line 3
      1 t_slice = slice("2025-07-29T23:53:20", "2025-07-29T23:53:30")
      2 dist_slice = slice(5.7e3, 6.e3)
----> 3 data = da.sel(distance=dist_slice, time=t_slice)
      4 data

File ~/miniconda3/envs/abyss/lib/python3.11/site-packages/xdas/core/dataarray.py:366, in DataArray.sel(self, indexers, method, endpoint, drop, **indexers_kwargs)
    364     indexers = {}
    365 indexers.update(indexers_kwargs)
--> 366 key = self.coords.to_index(indexers, method, endpoint)
    367 da = self[key]
    368 if drop:

File ~/miniconda3/envs/abyss/lib/python3.11/site-packages/xdas/core/coordinates.py:164, in Coordinates.to_index(self, item, method, endpoint)
    162 def to_index(self, item, method=None, endpoint=True):
    163     query = self.get_query(item)
--> 164     return {dim: self[dim].to_index(query[dim], method, endpoint) for dim in query}

File ~/miniconda3/envs/abyss/lib/python3.11/site-packages/xdas/core/coordinates.py:164, in <dictcomp>(.0)
    162 def to_index(self, item, method=None, endpoint=True):
    163     query = self.get_query(item)
--> 164     return {dim: self[dim].to_index(query[dim], method, endpoint) for dim in query}

File ~/miniconda3/envs/abyss/lib/python3.11/site-packages/xdas/core/coordinates.py:320, in Coordinate.to_index(self, item, method, endpoint)
    318 def to_index(self, item, method=None, endpoint=True):
    319     if isinstance(item, slice):
--> 320         return self.slice_indexer(item.start, item.stop, item.step, endpoint)
    321     else:
    322         return self.get_indexer(item, method)

File ~/miniconda3/envs/abyss/lib/python3.11/site-packages/xdas/core/coordinates.py:792, in InterpCoordinate.slice_indexer(self, start, stop, step, endpoint)
    790 if start is not None:
    791     try:
--> 792         start_index = self.get_indexer(start, method="bfill")
    793     except KeyError:
    794         start_index = len(self)

File ~/miniconda3/envs/abyss/lib/python3.11/site-packages/xdas/core/coordinates.py:778, in InterpCoordinate.get_indexer(self, value, method)
    776 except ValueError as e:
    777     if str(e) == "fp must be strictly increasing":
--> 778         raise ValueError(
    779             "overlaps were found in the coordinate. If this is due to some "
    780             "jitter in the tie values, consider smoothing the coordinate by "
    781             "including some tolerance. This can be done by "
    782             "`da[dim] = da[dim].simplify(tolerance)`, or by specifying a "
    783             "tolerance when opening multiple files."
    784         )
    785     else:
    786         raise e

ValueError: overlaps were found in the coordinate. If this is due to some jitter in the tie values, consider smoothing the coordinate by including some tolerance. This can be done by `da[dim] = da[dim].simplify(tolerance)`, or by specifying a tolerance when opening multiple files.

Attempting to simplify:

da["time"] = da["time"].simplify(tolerance=1e-2)

Which raises:

---------------------------------------------------------------------------
DTypePromotionError                       Traceback (most recent call last)
DTypePromotionError: The DType <class 'numpy.dtypes.TimeDelta64DType'> could not be promoted by <class 'numpy.dtypes.Float64DType'>. This means that no common DType exists for the given inputs. For example they cannot be stored in a single array unless the dtype is `object`. The full list of DTypes is: (<class 'numpy.dtypes.TimeDelta64DType'>, <class 'numpy.dtypes.Float64DType'>)

The above exception was the direct cause of the following exception:

UFuncTypeError                            Traceback (most recent call last)
Cell In[18], line 1
----> 1 da["time"] = da["time"].simplify(tolerance=1e-2)

File ~/miniconda3/envs/abyss/lib/python3.11/site-packages/xdas/core/coordinates.py:854, in InterpCoordinate.simplify(self, tolerance)
    852     else:
    853         tolerance = 0.0
--> 854 tie_indices, tie_values = douglas_peucker(
    855     self.tie_indices, self.tie_values, tolerance
    856 )
    857 return self.__class__(
    858     dict(tie_indices=tie_indices, tie_values=tie_values), self.dim
    859 )

File ~/miniconda3/envs/abyss/lib/python3.11/site-packages/xdas/core/coordinates.py:1034, in douglas_peucker(x, y, epsilon)
   1032 dmax = d[index]
   1033 index += start
-> 1034 if dmax > epsilon:
   1035     stack.append([start, index + 1])
   1036     stack.append([index, stop])

UFuncTypeError: ufunc 'greater' did not contain a loop with signature matching types (<class 'numpy.dtypes.TimeDelta64DType'>, <class 'numpy.dtypes.Float64DType'>) -> None

The xdas version is 0.2.3 (latest on pypi), numpy == 1.26.4

[atrabattoni]

Yes complex error indeed, but still it gives the correct hint on how to solve this issue. You are passing a float 1e-2 whereas the time coordinate is in datetime64[some_unit]. You should pass a timedelta64[any_unit] in that case, i.e.:

da["time"] = da["time"].simplify(tolerance=np.timedelta64(30, "ms"))

I agree that we could add a more friendly error an/or consider floats as seconds by default.

[martijnende]

The documentation seems to suggest that floats are expected, even though 0.0 is not super informative.

If the simplify method is assumed to be only applied to the temporal dimension, then before this line you could add:

elif isinstance(tolerance, float) or isinstance(tolerance, int):
  tolerance = np.timedelta64(int(tolerance * 1e3), "ms")

If this method needs to be generally applicable to any dimension (space, time, curvature), then it needs to inherit the unit from the parent class (don't know where that would be found).