nanoarrow integration

[eitsupi]

Ref: apache/arrow-nanoarrow#810

Given the importance of Apache Arrow, I think it's worth having nanoarrow as a required or optional dependency to automate the conversion from the Arrow C stream interface etc. on the Python side.

[t-kalinowski]

Sorry for the delay, and thanks for the thoughtful suggestion. I'm very happy to see the reticulate S3 methods land in nanoarrow with apache/arrow-nanoarrow#817!

We’re open in principle to wiring up something in reticulate that makes this smoother, but we’d want to avoid adding Arrow as a dependency (even in Suggests). Could you sketch a proposal that improves the R user experience while keeping Arrow off the dependency list?

[eitsupi]

We’re open in principle to wiring up something in reticulate that makes this smoother, but we’d want to avoid adding Arrow as a dependency (even in Suggests). Could you sketch a proposal that improves the R user experience while keeping Arrow off the dependency list?

It's not clear to me what you mean by "Arrow".

• If you are referring to the R arrow package, then it is completely unnecessary.
  (The reticulate integration has existed in the R arrow package for many years, but I would like to avoid using the R arrow package as much as possible, so I suggested using nanoarrow.)
• If you are referring to the nanoarrow package, could you please clarify why you would want to avoid it as it is very lightweight to install?
• If you're referring to the Python pyarrow package, I think pandas, which reticulate uses, already depends on pyarrow in many cases.

The reason I'm proposing this on the reticulate side instead of nanoarrow is because it allows for efficient translation between Python and R (which, as you know, I think Apache Arrow was started for).

For example, for a huge Pandas DataFrame with over 7 million rows, the speed difference between R conversion via nanoarrow and the current reticulate seems to be about 10 times on my machine.

library(reticulate)
library(nanoarrow)
library(tictoc)

py_require(c("pandas[pyarrow]==2.3.3", "nanoarrow==0.7.0"))

pd <- import("pandas", convert = FALSE)
na <- import("nanoarrow", convert = FALSE)

pdf <- pd$read_parquet(
  "https://github.com/duckdb/duckdb-data/releases/download/v1.0/taxi_2019_04.parquet"
)

tic()
py_to_r(pdf) |>
  tibble::as_tibble()
#> # A tibble: 7,433,139 × 18
#>    vendor_id pickup_at           dropoff_at          passenger_count
#>    <chr>     <dttm>              <dttm>                        <int>
#>  1 1         2019-04-01 00:04:09 2019-04-01 00:06:35               1
#>  2 1         2019-04-01 00:22:45 2019-04-01 00:25:43               1
#>  3 1         2019-04-01 00:39:48 2019-04-01 01:19:39               1
#>  4 1         2019-04-01 00:35:32 2019-04-01 00:37:11               1
#>  5 1         2019-04-01 00:44:05 2019-04-01 00:57:58               1
#>  6 1         2019-04-01 00:29:16 2019-04-01 00:38:00               1
#>  7 1         2019-04-01 00:06:47 2019-04-01 00:08:15               1
#>  8 1         2019-04-01 00:52:16 2019-04-01 00:55:10               1
#>  9 2         2019-04-01 00:52:28 2019-04-01 01:11:24               1
#> 10 1         2019-04-01 00:02:19 2019-04-01 00:03:05               1
#> # ℹ 7,433,129 more rows
#> # ℹ 14 more variables: trip_distance <dbl>, rate_code_id <chr>,
#> #   store_and_fwd_flag <chr>, pickup_location_id <int>,
#> #   dropoff_location_id <int>, payment_type <chr>, fare_amount <dbl>,
#> #   extra <dbl>, mta_tax <dbl>, tip_amount <dbl>, tolls_amount <dbl>,
#> #   improvement_surcharge <dbl>, total_amount <dbl>, congestion_surcharge <dbl>
toc()
#> 39.777 sec elapsed

tic()
na$Array(pdf) |>
  as_nanoarrow_array_stream() |>
  tibble::as_tibble()
#> # A tibble: 7,433,139 × 18
#>    vendor_id pickup_at           dropoff_at          passenger_count
#>    <chr>     <dttm>              <dttm>                        <int>
#>  1 1         2019-04-01 00:04:09 2019-04-01 00:06:35               1
#>  2 1         2019-04-01 00:22:45 2019-04-01 00:25:43               1
#>  3 1         2019-04-01 00:39:48 2019-04-01 01:19:39               1
#>  4 1         2019-04-01 00:35:32 2019-04-01 00:37:11               1
#>  5 1         2019-04-01 00:44:05 2019-04-01 00:57:58               1
#>  6 1         2019-04-01 00:29:16 2019-04-01 00:38:00               1
#>  7 1         2019-04-01 00:06:47 2019-04-01 00:08:15               1
#>  8 1         2019-04-01 00:52:16 2019-04-01 00:55:10               1
#>  9 2         2019-04-01 00:52:28 2019-04-01 01:11:24               1
#> 10 1         2019-04-01 00:02:19 2019-04-01 00:03:05               1
#> # ℹ 7,433,129 more rows
#> # ℹ 14 more variables: trip_distance <dbl>, rate_code_id <chr>,
#> #   store_and_fwd_flag <chr>, pickup_location_id <int>,
#> #   dropoff_location_id <int>, payment_type <chr>, fare_amount <dbl>,
#> #   extra <dbl>, mta_tax <dbl>, tip_amount <dbl>, tolls_amount <dbl>,
#> #   improvement_surcharge <dbl>, total_amount <dbl>, congestion_surcharge <dbl>
toc()
#> 3.194 sec elapsed

sessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value
#>  version  R version 4.5.1 (2025-06-13)
#>  os       Ubuntu 24.04.3 LTS
#>  system   x86_64, linux-gnu
#>  ui       X11
#>  language (EN)
#>  collate  en_US.UTF-8
#>  ctype    en_US.UTF-8
#>  tz       Etc/UTC
#>  date     2025-10-25
#>  pandoc   3.7.0.2 @ /usr/bin/ (via rmarkdown)
#>  quarto   NA
#>
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package     * version     date (UTC) lib source
#>  cli           3.6.5       2025-04-23 [1] RSPM
#>  digest        0.6.37      2024-08-19 [1] RSPM
#>  evaluate      1.0.5       2025-08-27 [1] RSPM
#>  fastmap       1.2.0       2024-05-15 [1] RSPM
#>  fs            1.6.6       2025-04-12 [1] RSPM
#>  glue          1.8.0       2024-09-30 [1] RSPM
#>  here          1.0.2       2025-09-15 [1] RSPM
#>  htmltools     0.5.8.1     2024-04-04 [1] RSPM
#>  jsonlite      2.0.0       2025-03-27 [1] RSPM
#>  knitr         1.50        2025-03-16 [1] RSPM
#>  lattice       0.22-7      2025-04-02 [2] CRAN (R 4.5.1)
#>  lifecycle     1.0.4       2023-11-07 [1] RSPM
#>  magrittr      2.0.4       2025-09-12 [1] RSPM
#>  Matrix        1.7-3       2025-03-11 [2] CRAN (R 4.5.1)
#>  nanoarrow   * 0.7.0.9000  2025-10-25 [1] Github (apache/arrow-nanoarrow@464bd7c)
#>  pillar        1.11.1      2025-09-17 [1] RSPM
#>  pkgconfig     2.0.3       2019-09-22 [1] RSPM
#>  png           0.1-8       2022-11-29 [1] RSPM
#>  rappdirs      0.3.3       2021-01-31 [1] RSPM
#>  Rcpp          1.1.0       2025-07-02 [1] RSPM
#>  reprex        2.1.1       2024-07-06 [1] RSPM
#>  reticulate  * 1.43.0.9001 2025-10-25 [1] Github (rstudio/reticulate@ef7e945)
#>  rlang         1.1.6       2025-04-11 [1] RSPM
#>  rmarkdown     2.29        2024-11-04 [1] RSPM
#>  rprojroot     2.1.1       2025-08-26 [1] RSPM
#>  sessioninfo   1.2.3       2025-02-05 [1] RSPM
#>  tibble        3.3.0       2025-06-08 [1] RSPM
#>  tictoc      * 1.2.1       2024-03-18 [1] RSPM
#>  utf8          1.2.6       2025-06-08 [1] RSPM
#>  vctrs         0.6.5       2023-12-01 [1] RSPM
#>  withr         3.0.2       2024-10-28 [1] RSPM
#>  xfun          0.53        2025-08-19 [1] RSPM
#>  yaml          2.3.10      2024-07-26 [1] RSPM
#>
#>  [1] /usr/local/lib/R/site-library
#>  [2] /usr/local/lib/R/library
#>  * ── Packages attached to the search path.
#>
#> ─ Python configuration ───────────────────────────────────────────────────────
#>  python:         /home/username/.cache/R/reticulate/uv/cache/archive-v0/2hZNU2hrrXZjh4S9DVDv8/bin/python
#>  libpython:      /home/username/.cache/R/reticulate/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/libpython3.12.so
#>  pythonhome:     /home/username/.cache/R/reticulate/uv/cache/archive-v0/2hZNU2hrrXZjh4S9DVDv8:/home/username/.cache/R/reticulate/uv/cache/archive-v0/2hZNU2hrrXZjh4S9DVDv8
#>  virtualenv:     /home/username/.cache/R/reticulate/uv/cache/archive-v0/2hZNU2hrrXZjh4S9DVDv8/bin/activate_this.py
#>  version:        3.12.12 (main, Oct 14 2025, 21:25:31) [Clang 20.1.4 ]
#>  numpy:          /home/username/.cache/R/reticulate/uv/cache/archive-v0/2hZNU2hrrXZjh4S9DVDv8/lib/python3.12/site-packages/numpy
#>  numpy_version:  2.3.4
#>  pandas:         /home/username/.cache/R/reticulate/uv/cache/archive-v0/2hZNU2hrrXZjh4S9DVDv8/lib/python3.12/site-packages/pandas
#>
#>  NOTE: Python version was forced by py_require()
#>
#> ──────────────────────────────────────────────────────────────────────────────

^{Created on 2025-10-25 with reprex v2.1.1}

[eddelbuettel]

nanoarrow is (at least in my eyes) a bit of a funny package in that it "contains two packages inside". First, nanoarrow contains a perfectly self-contained C++ library you could vendor to convert to/from Arrow objects. With zero Arrow dependencies, as this is the point of the C API offered by Arrow and implemented by this library. Second, there is also an R package for conveniently creating, inspecting, consuming, ... (nano)arrow objects in R.

As reticulate stands in the middle between Python and R and users may already have Arrow capabilities on either end (say: from/to dplyr to/from pandas) we may not need the R bits. Part of the discussion arose in the ticket over at nanoarrow that @eitsupi referenced at the very top, and part of what it does is now easier thanks to the recent change there that you reference. We may need @paleolimbot in here to best explain how this

[paleolimbot]

Thanks all for the discussion here...a few years ago I redid the (macro)Arrow reticulate integration and was happy to add the nanoarrow integration...reticulate has come a long way since then and it was fun!

nanoarrow is (at least in my eyes) a bit of a funny package

Agreed 🙂 . I think the most useful thing it does is provide the as_nanoarrow_array/schema/array_stream() generics. It's like a lightweight interchange package so that nobody has to put (macro)Arrow (even in Suggests). It also has converters to/from R objects, which is a bit of a separate scope but is kind of essential to help implement those generics.

That PR into nanoarrow basically implements as_nanoarrow_array/schema/array_stream() generics for arbitrary (wrapped) Python objects. I'm not sure that has a better equivalent if it lived in reticulate...probably the only thing reticulate could do to help is let me create named PyCapsules (because then I could implement it without nanoarrow or any other package on the Python side).

From the Python side, it may be helpful (or more magical) to have "R wrapper" objects implement __arrow_c_schema__, __arrow_c_array__, and/or __arrow_c_array_stream__ (which are the Python equivalent of R/nanoarrow's generics). I'm also not sure exactly how useful that is since as far as I can tell that conversion happens eagerly (i.e., by the time it hits Python it's already been constructed as a Python object which does or doesn't implement the interface). I'm not sure how many objects that would be wrapped in an R Capsule can or want to implement that interface; however, the support for the Arrow PyCapsule interface in the Python ecosystem is getting to be very good such that you might be able to defer conversion by (e.g.) sending DataFrames over as unconverted wrapped objects that implement the interface. In that scenario you could dispatch to as_nanoarrow_array/schema/array_stream() when those dunder methods are called.

[eitsupi]

I tried implementing some conversions between Python Polars and R Polars as a test (pola-rs/r-polars#1607).

The current conversion from a Python Polars DataFrame to an R data.frame, as implemented in #1568, requires pyarrow and pandas on the Python side, but using nanoarrow only requires nanoarrow on both the Python and R sides.
The latter way results in significantly smaller package sizes and is much faster.