R versions to support and which to cover in test suite

[clhunsen]

Description

Originating in #152, the question arose which R versions we should support and which to cover in the test suite.

Motivation

Currently, we explicitly support R version 3.3.1 (see README.md). However, new versions of R are released over time and we should ensure that these work with our network library in principle.

Ideas for the Implementation

There are two work items for this issue (see also the discussion in #152):

1. We definitely need to cover version 3.3.x in the test suite (i.e., TravisCI), since this version is installed in all our containers!
2. We need to find a way to reliably and correctly install all dependencies (i.e., R packages) for all R version covered in our .travis.yml. Some dependencies are not available for older R version anymore. See TravisCI tests are failing for R 3.3 #152 (comment) for some further details.

---

What are your thoughts? Is this an issue that can be stalled until we transform this project itself into a proper R package?

[bockthom]

Currently (for about two weeks), our builds on travis ci fail for R versions 3.3, 3.4, and 3.5. Only 3.6 builds correctly. I had a look at the build logs and identified the following:
(1) For R versions 3.3, 3.4, and 3.5, the build is terminated as the maximum log size is exceeded.
(2) Why is the log size exceeded? It's mostly due to installing lots of R packages and errors during installing them. It looks to me that also R packages are installed that are never used (also not as a dependency, to the best of my knowledge), which fail due to missing or wrong dependencies. On the other hand, some needed R packages cannot be installed any more as they are not available for outdated R versions any more.

As all builds on all branches fail due to the above described problems, I tried to figure out whether there is an easy option on how to fix the problems. To reduce the log size, we could set log level to INFO instead of DEBUG. However, this won't help us in general when the verbose log of installing packages is the reason for exceeding the maximum log length. So, the more important problem is dealing with installing the packages. Unfortunately, I did not find acceptable solutions. It still seems weird to me that two weeks ago everything worked for all R versions 3.3, 3.4, 3.5 and now it does not work for neither of them.

I also had a look at the solutions provided in #152. Most promising approach was the oldr::install.compatible.packages function. However, even installing oldr is not possible for outdated R versions any more - only option would be to pin oldr to a specific version, because then we won't get new versions of this package even for recent R versions. But this also introduces new dependencies which are not necessary, which I really dislike.

As I would like to see green check-marks next to our build logs again, the only practicable solution for now would be to restrict test runs to R version 3.6 (as 3.6.3 is the R version our current container setups use). Later we should also support R version 4, but there may be some braking changes, so I won't give it a try yet. However, even when removing R 3.3, 3.4, 3.5 from the test runs, we might result in a similar problem for R version 3.6 soon. Hence, this will only solve the problem for the short term. Automatically installing a compatible version would be great, why can't R do that automatically?

@clhunsen What is your opinion on that?

[clhunsen]

My personal Travis builds

Interestingly, the weekly Travis build for my fork ran successfully just yesterday. 🤔 Maybe, the extra output of the new dependencies (introduced in PR #173) is too much. Maybe, the reason lies somewhere else.

Thank you very much for your insights and thoughts on this, @bockthom. My thoughts and findings follow.

Reducing the log length

I thought about the problem in the last days too and found a possible workaround that may help to reduce the log length: we should install packages with reduced verbosity. The output of the actual tests only makes up ~10k lines, while the package installation makes up over 30k lines, stressing the maximum log length.

I tested some approaches and found that adapting the call to install.packages(...) in install.R directly is the best approach to (temporarily) reduce the log length.

install.packages(..., quiet = TRUE, verbose = TRUE)

The output prints the progress, but does not log any compilation output to the console. Maybe, this helps in the search for problems with the Travis builds.

Output looks like this (for a failing installation):

> install.packages("udunits2", quiet = TRUE, verbose = TRUE)
system (cmd0): /usr/lib64/R/bin/R CMD INSTALL
foundpkgs: udunits2, /tmp/RtmpihNqb0/downloaded_packages/udunits2_0.13.tar.gz
files: /tmp/RtmpihNqb0/downloaded_packages/udunits2_0.13.tar.gz
Warning message:
In install.packages("udunits2", quiet = TRUE, verbose = TRUE) :
  installation of package ‘udunits2’ had non-zero exit status

R 4.0

I have R 4.0 installed on my notebook, the installation of packages passed without any problems, and the test suite of coronet passed completely. So, support for R 4.0 should be no problem, I guess. 🙂

Docker

Maybe, the creation of a Docker container for using coronet in a safe and supported environment is a long-term solution for both adopters/users and Travis tests. 🤔 This would also lay a good basis for controlled updates of coronet, R version, dependencies, system, ... Just an idea. 🤷‍♂️

[bockthom]

Thanks for your answer @clhunsen. I have a few comments on it:

My personal Travis builds

Interestingly, the weekly Travis build for my fork ran successfully just yesterday. thinking Maybe, the extra output of the new dependencies (introduced in PR #173) is too much.

Nope. This is not related to PR #173. The travis build fails also for the current dev and master branches which are unchanged since February 2020, and which had successful builds until two weeks ago. (And even #173 had successful builds until two weeks ago, and there has essentially nothing changed in this PR since then.) Hence, the new dependencies introduced in #173 cannot have caused the problem.

The difference between the builds on your fork and the builds on the main project is: You still use cached package installations, the main project has to rebuild the package installations. I don't know what has triggered that, but rebuilding the packages is not possible any more and results in lots of errors bloating the log. I'll promise that your fork will also result in build failures if the packages would have to been rebuilt. You're just in luck that you still can rely on travis's cached package installations.

And reinstalling the packages also installs lots of packages that are never used, I don't know why. Most of these packages result in compile errors. So, eventually, I am not sure if just reducing the log length will suffice to get green check-marks in the end, as those package installations may result in weird behavior...

Reducing the log length

I thought about the problem in the last days too and found a possible workaround that may help to reduce the log length: we should install packages with reduced verbosity. The output of the actual tests only makes up ~10k lines, while the package installation makes up over 30k lines, stressing the maximum log length.

True. But most of those over 30k lines are a result of the failing package installations. Normal, succeeding package installations result in a smaller log output (but may still make up more lines than the actual tests).

I tested some approaches and found that adapting the call to install.packages(...) in install.R directly is the best approach to (temporarily) reduce the log length.

install.packages(..., quiet = TRUE, verbose = TRUE)

Sounds good! Thanks for thinking about that and testing this solution. Independent from the current issue, this will help us for the future when adding new tests may bring us closer to the maximum log length in general. So, it's definitely worth trying to be well positioned for future enhancements.

R 4.0

I have R 4.0 installed on my notebook, the installation of packages passed without any problems, and the test suite of coronet passed completely. So, support for R 4.0 should be no problem, I guess. slightly_smiling_face

Short answer: Sounds good! Thanks for testing that. So, whenever touching the travis configuration next time, I will add R 4.0 to the R versions used for the builds and tests.

Docker

Maybe, the creation of a Docker container for using coronet in a safe and supported environment is a long-term solution for both adopters/users and Travis tests. thinking This would also lay a good basis for controlled updates of coronet, R version, dependencies, system, ... Just an idea. man_shrugging

Good idea. But it is costly in terms of preparation time (which we don't have at the moment), and which does not help us with the issues we are facing at the moment, i.e., figuring out why package installations fail (or, to be precise why the packages have to be rebuilt and there are lots of package installations failing now which never have been installed or failed before...) But I will keep the Docker idea in mind for future changes.