Failed HTTP request when using get_vpts

[YohanSassi]

I regularly encountered the same error while using getRad::get_vpts in a shiny app developed to flag rain contaminations in the European weather radar data:

# download the data of a radar, from the Aloft repository, in folder "baltrad" for the period of interest
  vpts_raw <- get_vpts(radar = wantRadar,
                       datetime = wantInterTime,
                       source = "baltrad")

"Listening on http://127.0.0.1:3807
Warning: Error in get_vpts: ℹ In index: 1.
Caused by error in httr2::req_perform_parallel():
! Failed to perform HTTP request.
Caused by error:
! Timeout was reached [aloftdata.s3-eu-west-1.amazonaws.com]:
Connection timed out after 10001 milliseconds
82:
81: signalCondition
80: signal_abort
79: rlang::abort
78: cli::cli_abort
77:
76: signalCondition
75: signal_abort
74: cnd_signal
73: httr2::req_perform_parallel
72: fetch_from_url_raw
68: read_vpts_from_url
59: get_vpts_aloft
58: .f
54: map_
53: purrr::map
50: get_vpts
49: server [#4]
3: runApp
2: print.shiny.appobj
1:
Error in get_vpts(radar = wantRadar, datetime = wantInterTime, source = "baltrad") :
ℹ In index: 1.
Caused by error in httr2::req_perform_parallel():
! Failed to perform HTTP request.
Caused by error:
! Timeout was reached [aloftdata.s3-eu-west-1.amazonaws.com]:
Connection timed out after 10001 milliseconds"

Unfortunately, I can't provide a replicable example as it sometimes works the first time and sometimes I get this error. When this happens, I resend the command (several times) and it finally works. The problem arises when looping through several radars/years and loading data iteratively for specific periods. If the connection to Aloft fails, then the whole loop crashes. Is there a way to force get_vpts to retry the connection if this error appears, and/or modify the connection time threshold after which the request is considered failed (to avoid the entire loop crashing)?

[bart1]

I have seen similar error come by in testing. Maybe the retries here:

getRad/R/utils.R

Line 425 in 4b84a26

purrr::map(req_retry_getrad) |>

are not working correctly. @PietrH do you know a way to test this. Or should I just try spamming aloft to test?

[bart1]

Just lot of calls in parallel does not seem to do the job (to recreate the error):

devtools::load_all("~/getRad")
#> ℹ Loading getRad

a<-glue::glue("https://aloftdata.s3-eu-west-1.amazonaws.com/baltrad/daily/bejab/2024/bejab_vpts_{strftime(as.Date('2024-2-6')+1:30,'%Y%m%d')}.csv") |> 
  getRad:::read_vpts_from_url()

^{Created on 2025-10-08 with reprex v2.1.1}

[PietrH]

For testing it would make sense to inject a request that guarantees a timeout or other http error code. We could also increase the timeout threshold by default to over 10 seconds. Something like http://httpbin.org/delay/15

Retries in parallel requests behave a little strange. It might be worth testing how much speed we actually lose by switching to sequential requests just so we can have more consistent retry behaviour.

See https://httr2.r-lib.org/reference/req_perform_parallel.html

[bart1]

I would think the default timeout of 10 seconds should be fine for serving a csv. If were going to fail I would prefer it not to take too long. Especially if you are going to multiply the timeout with multiple retries it could be minutes before failing.

With respect to retry and parallel that would be good to investigate. From the manual it sound that it should still retry just in a particular way. I just tried a three R sessions at the same time and I'm still not able to trigger the issue. Would you be able to investigate/resolve this?

[PietrH]

I'm about to be away on conference until the mid of November, so it would be a while before I can get back to you on this.

[bart1]

I'm about to be away on conference until the mid of November, so it would be a while before I can get back to you on this.

Ok lets see if we can reach a solution before that. Otherwise I'll leave it open.

@YohanSassi could you give more information on the conditions that this occurs, being able to reproduce it would be hugely helpful.

[YohanSassi]

Writing this really simple code end up creating the error:

RadarNames <- c("deros","sekaa","sehem","seatv","sebaa","sehuv","seoer","sella","fiuta","fivim","fikor","eehar")
  
for(i in RadarNames){
wantInterTime <- interval(ymd_hms("2024-08-01 00:00:00"),
                          ymd_hms("2024-08-30 23:59:59"))

vpts_raw <- get_vpts(radar = RadarNames, 
                         datetime = wantInterTime,
                         source = "baltrad")
}


Error in `get_vpts()`:
ℹ In index: 7.
Caused by error in `httr2::req_perform_parallel()`:
! Failed to perform HTTP request.
Caused by error:
! Timeout was reached [aloftdata.s3-eu-west-1.amazonaws.com]:
Connection timed out after 10002 milliseconds
Run `rlang::last_trace()` to see where the error occurred.

[bart1]

Thanks a lot, I just tried to run the code (btw reprex is a really nice package for copy pasting results into github).

Most of the times I run the code below it works fine for me I hover did once get the error

require(getRad)
#> Loading required package: getRad
RadarNames <- c("deros","sekaa","sehem","seatv","sebaa","sehuv","seoer","sella","fiuta","fivim","fikor","eehar")
wantInterTime <- lubridate::interval(lubridate::ymd_hms("2024-08-01 00:00:00"),
                                     lubridate::ymd_hms("2024-08-30 23:59:59"))
vpts_raw <- get_vpts(radar = RadarNames, 
                     datetime = wantInterTime,
                     source = "baltrad")
for(i in RadarNames){
  vpts_raw <- get_vpts(radar = i, 
                       datetime = wantInterTime,
                       source = "baltrad")
}

^{Created on 2025-10-09 with reprex v2.1.1}

[bart1]

@YohanSassi what kind of operating system and internet connection do you use? that might influence the frequency of the error

[YohanSassi]

True I did not thought about that, I'm on macOS - and on the Lund University Wifi. I will try to find a way to test with a wired internet connection.

[bart1]

Hmm I assume Lund University has decent wifi, it would be good to test but I was mainly wondering if it was something like adsl which does make a big difference