Randomly appearing bug with multi-threaded Transformer in versions >= 3.7.0 #1499
Description
Code Sample, a copy-pastable example if possible
import numpy as np
from pyproj import Transformer
from joblib import Parallel, delayed
import time
# Generate test data
N = 1000
lat = np.random.uniform(-90, 90, N)
lon = np.random.uniform(-180, 180, N)
alt = np.random.uniform(0, 1000, N)
composite_crs = "EPSG:4326+3855"
ECEF_crs = "EPSG:4978"
chunk_size = 128
# Split into chunks
chunks = [
(lon[i:i+chunk_size], lat[i:i+chunk_size], alt[i:i+chunk_size])
for i in range(0, N, chunk_size)
]
def run_single_threaded():
tf = Transformer.from_crs(composite_crs, ECEF_crs, always_xy=True)
results = tf.transform(lon, lat, alt)
return results
def run_multi_threaded():
def transform_chunk(lon_chunk, lat_chunk, alt_chunk):
tf = Transformer.from_crs(composite_crs, ECEF_crs, always_xy=True)
return tf.transform(lon_chunk, lat_chunk, alt_chunk)
results = Parallel(n_jobs=-1, prefer="threads")(
delayed(transform_chunk)(lon, lat, alt) for lon, lat, alt in chunks
)
return np.concatenate([r[0] for r in results]), np.concatenate([r[1] for r in results]), np.concatenate([r[2] for r in results])
x1, y1, z1 = run_single_threaded()
x2, y2, z2 = run_multi_threaded()
# --- Check results
def check_close(a, b, label):
if np.allclose(a, b, atol=1e-6):
print(f"{label:20s}: Match")
else:
print(f"{label:20s}: MISMATCH")
check_close(x1, x2, "Single vs Threads (X)")
check_close(y1, y2, "Single vs Threads (Y)")
check_close(z1, z2, "Single vs Threads (Z)")Problem description
After switching to 3.7.0, converting coordinates to a composite CRS EPSG:4326+3855 while multi-threading sometimes throws results with chunks of inf values.
Here is the output of the above code with versions >= 3.7.0
TIFFFillTile: Read error at row 2816, col 2816, tile 54; got 0 bytes, expected 124661.
TIFFFillTile: Read error at row 4352, col 0, tile 296; got 0 bytes, expected 137296.
TIFFFillTile: Read error at row 4352, col 0, tile 317; got 0 bytes, expected 166622.
TIFFFillTile: Read error at row 2560, col 2560, tile 351; got 0 bytes, expected 160283.
Single vs Threads (X): MISMATCH
Single vs Threads (Y): MISMATCH
Single vs Threads (Z): MISMATCH
Note that is does not occur every time but randomly.
Expected Output
With versions 3.6.1 it seems to be always running as expected:
Single vs Threads (X): Match
Single vs Threads (Y): Match
Single vs Threads (Z): Match
Environment Information
pyproj info:
pyproj: 3.7.0
PROJ (runtime): 9.5.1
PROJ (compiled): 9.5.0
data dir: /root/micromamba/envs/eo_tools/share/proj
user_data_dir: /root/.local/share/proj
PROJ DATA (recommended version): 1.20
PROJ Database: 1.4
EPSG Database: v11.022 [2024-11-05]
ESRI Database: ArcGIS Pro 3.4 [2024-11-04]
IGNF Database: 3.1.0 [2019-05-24]
System:
python: 3.12.10 | packaged by conda-forge | (main, Apr 10 2025, 22:21:13) [GCC 13.3.0]
executable: /root/micromamba/envs/eo_tools/bin/python
machine: Linux-6.8.0-59-generic-x86_64-with-glibc2.36
Python deps:
certifi: 2025.4.26
Cython: None
setuptools: 80.9.0
pip: 25.1.1
Installation method
conda (actually micromamba) using the conda-forge repo.
Conda environment information (if you installed with conda):
Environment (
conda list):
Details
$ conda list proj
proj 9.5.1 h0054346_0 conda-forge
pyproj 3.7.0 py312he630544_0 conda-forge
Details about
conda and system ( conda info ):
Details
$ conda info
libmamba version : 2.1.1
micromamba version : 2.1.1
curl version : libcurl/8.13.0 OpenSSL/3.5.0 zlib/1.3.1 zstd/1.5.7 libssh2/1.11.1 nghttp2/1.64.0
libarchive version : libarchive 3.7.7 zlib/1.3.1 bz2lib/1.0.8 libzstd/1.5.7
envs directories : /root/micromamba/envs
package cache : /root/micromamba/pkgs
/root/.mamba/pkgs
environment : eo_tools (active)
env location : /root/micromamba/envs/eo_tools
user config files : /root/.mambarc
populated config files :
virtual packages : __unix=0=0
__linux=6.8.0=0
__glibc=2.36=0
__archspec=1=x86_64_v3
channels : https://conda.anaconda.org/conda-forge/linux-64
https://conda.anaconda.org/conda-forge/noarch
base environment : /root/micromamba
platform : linux-64