encode64 creates keys that are too long sometimes

[kendonB]

While using drake I found the error:

Error in file.rename(tmp, filename) : expanded 'to' name too long

and dug in to find that it is ultimately caused by R or Windows getting upset that it couldn't create a file with such a big name. Is there any modification for encode64 that could limit the size of these names somehow?

cc @wlandau

[wlandau]

Also related: #88, #90.

[billdenney]

I just had the same issue that in windows a 180-character filename was attempted to be created. When I went into the same directory and tried to create a file from Windows explorer (the normal way to browse directories), it only allowed a 146-character filename. (This was during an attempted drake::make() call.)

The error occurred here:

storr/R/hash.R

Line 124 in 2750823

file.rename(tmp, filename)

I thought that the following would fix the issue since md5 keys are relatively short, but it does not appear to:

my_cache <- storr::storr_rds(path=".drake/", hash_algorithm="md5", mangle_key=FALSE)

[wlandau]

Yeah, this one stumped me. I do not have a solution.

However, I can tell you that the keys in this issue are different from md5 keys. Hash functions like md5 serve to fingerprint the data objects, e.g. the return values of targets. The keys we are talking about here are things like target names and file names.

Since version 7.0.0, drake no longer mangles all the keys. Now, it only mangles namespaced functions names (i.e. pkg::fun()) and file paths, and it uses base64url::base64_urlencode() rather than storr::encode64(). So one thing you could do is try to reduce the length of your file paths or reduce reliance on custom input/output files.

[billdenney]

My work-around was to put my cache into c:/tmp/a/.drake, but that seems like a kludge. I don't readily have control over the input filenames (they come from a client).

I'm not familiar with the full structure used by the cache, but it looks like it is creating a lot of small files that simply contain hashes. Would it be reasonable to replace the objects directory with an objects.RDS file that is a list of the hashes instead of a bunch of small files?

Alternatively, could the p-... files be named something like:
p-
Then the contents are pairs of lines that are the full filename and the md5 hash of the file

Since there is a potential clash at the md5 hash of the full path, the format of the file would need to be different.

Another more radical idea, would be to encode to Unicode using a custom encoder. That would shorten anything (and it could be humorous if you used something like https://github.com/tsunaminoai/baseEmoji). :) That should work on Windows and Linux.

[wlandau]

I'm not familiar with the full structure used by the cache, but it looks like it is creating a lot of small files that simply contain hashes. Would it be reasonable to replace the objects directory with an objects.RDS file that is a list of the hashes instead of a bunch of small files?

Unfortunately, that would slow things down (you would need to read the entire objects.RDS every time you want to change it). The next best thing is a true database (e.g. storr_dbi()). But either way, we would incur a parallel bottleneck. The nice thing about storr_rds() and its swarm of tiny files is that it supports parallel writes.

Then the contents are pairs of lines that are the full filename and the md5 hash of the file

Those hashes (md5, drake's default xxhash64, or otherwise) are stored inside the key files themselves, not in their file names. For the key file names, we need an invertible transformation so we can get those file paths back. Base 64 encoding is invertible, but hash functions are not.

Another more radical idea, would be to encode to Unicode using a custom encoder. That would shorten anything (and it could be humorous if you used something like tsunaminoai/baseEmoji). :) That should work on Windows and Linux.

That sounds really cool, and I am not sure it has been tried (need to read up on that). The encoding would certainly be far more compact than the base 64 version. But to be honest, I am skeptical because our options for cross-platform characters are severely limited even with ASCII. And it would be a breaking change for storr_rds().

[billdenney]

Jumping right to the end, I understand that it would be a breaking change for storr_rds(), but it would help us unfortunate Windows users. :)

A way to make current caches forward compatible would be for it to look for the current style file first, and if found read it, delete it, and write the new style (maybe just do that on load so that it is not a check with every access). It would break backward compatibility, though.

As for compatibility, a more complete source from Microsoft suggests that the modern Windows storage occurs with Unicode allowed, so it would work there: https://docs.microsoft.com/en-us/windows/desktop/fileio/naming-a-file

The Wikipedia page on file names suggests that it would work on Linux and OSX: https://en.m.wikipedia.org/wiki/Filename#Unicode_interoperability

I’m not certain, but I think that the answer for Solaris is the same as for Linux.

[billdenney]

It appears that we could make a base-146 encoding system, and I think that the main limitation is R and not Windows for limiting us to 146 characters.

I just tested all UTF-8 characters from 0x1 to 0xff00 on Windows, and the following 146 characters work to create unique, single-character filenames that are not overwritten. ```
!#$%&'()+,-0123456789;=@abcdefghijklmnopqrstuvwxyz[]^_`{}~ ¡¢£¤¥¦§¨©ª«¬®¯°±²³´µ¶·¸¹º»¼½¾¿ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖ×ØÙÚÛÜÝÞß÷ÿŒŠŽƑˆ˜–—‘’‚“”„†‡•…‰‹›€™

All of these also worked to create single-character filenames on Linux that were unique.

The main issue on Windows appears to be the R interface for naming files doesn't like Unicode since the errors tended to look like the following, and I was able to create more files using other Windows interfaces.

11839 failed
Error in file(con, "w") : cannot open the connection
In addition: Warning message:
In file(con, "w") :
cannot open file 'c:/tmp/filenametest/<U+FEF9>': Invalid argument

I used the code below for testing in Windows

```r
library(utf8)

unicode_characters <- sapply(X=0x1:0xff00, FUN=intToUtf8)
unicode_widths <- sapply(X=unicode_characters, FUN=utf8_width)
one_width_characters <-
  setdiff(
    unicode_characters[unicode_widths %in% 1],
    c("\\", "/", ":", "*", "?", '"', "<", ">", "|") # disallowed characters in Windows
  )


base_path <- "c:/tmp/filenametest"
dir.create(path=base_path)
failed_characters <- c()
overwritten_characters <- c()

for (current_char in one_width_characters) {
  filename <- file.path(base_path, current_char)
  success <- FALSE
  n_files_start <- length(list.files(path=base_path))
  try({
    writeLines(text=current_char, con=filename)
    success <- TRUE
  })
  if (!success) {
    message(current_char, " failed")
    failed_characters <- c(failed_characters, current_char)
  } else {
    n_files <- length(list.files(path=base_path)) - n_files_start
    if (n_files != 1) {
      message(current_char, " overwrote")
      overwritten_characters <- c(overwritten_characters, current_char)
    }    
  }
}

And this variant for Linux:

good_windows_characters <-
  "!#$%&'()+,-0123456789;=@ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_`{}~ ¡¢£¤¥¦§¨©ª«¬®¯°±²³´µ¶·¸¹º»¼½¾¿ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖ×ØÙÚÛÜÝÞß÷ÿŒŠŽƑˆ˜–—‘’‚“”„†‡•…‰‹›€™"
good_windows_characters_l <- strsplit(good_windows_characters, split="")[[1]]

base_path <- "~/tmp/filenametest"
dir.create(path=base_path)
failed_characters <- c()
overwritten_characters <- c()

for (current_char in good_windows_characters_l) {
  filename <- file.path(base_path, current_char)
  success <- FALSE
  n_files_start <- length(list.files(path=base_path))
  try({
    writeLines(text=current_char, con=filename)
    success <- TRUE
  })
  if (!success) {
    message(current_char, " failed")
    failed_characters <- c(failed_characters, current_char)
  } else {
    n_files <- length(list.files(path=base_path)) - n_files_start
    if (n_files != 1) {
      message(current_char, " overwrote")
      overwritten_characters <- c(overwritten_characters, current_char)
    }    
  }
}

[billdenney]

It turns out that I found a bug in the file.path() function in Windows (https://bugs.r-project.org/bugzilla/show_bug.cgi?id=17583). If I use paste0(), a much larger universe of characters is usable in Windows (10895 characters appear to work, RDS save in ascii format is attached; read with readRDS()). This could cut the filename length by about 3-fold (13.4 bits per character instead of 5), depending on how characters are counted.

good_characters.txt

The new Windows code is:

library(utf8)

unicode_characters <- sapply(X=0x1:0xff00, FUN=intToUtf8)
unicode_widths <- sapply(X=unicode_characters, FUN=utf8_width)
one_width_characters <-
  setdiff(
    unicode_characters[unicode_widths %in% 1],
    c("\\", "/", ":", "*", "?", '"', "<", ">", "|") # disallowed characters in Windows
  )


base_path <- "c:/tmp/filenametest"
dir.create(path=base_path)
failed_characters <- c()
overwritten_characters <- c()

for (current_char in one_width_characters) {
  filename <- paste0(base_path, "/", current_char)
  success <- FALSE
  n_files_start <- length(list.files(path=base_path))
  try({
    writeLines(text="A", con=filename)
    success <- TRUE
  })
  if (!success) {
    message(current_char, " failed")
    failed_characters <- c(failed_characters, current_char)
  } else {
    n_files <- length(list.files(path=base_path)) - n_files_start
    if (n_files != 1) {
      message(current_char, " overwrote")
      overwritten_characters <- c(overwritten_characters, current_char)
    }    
  }
}

good_char <-
  setdiff(one_width_characters, c(failed_characters, overwritten_characters))
saveRDS(object=good_char, file="good_characters.txt", ascii=TRUE)

The new Linux code is:

good_windows_characters_l <- readRDS("good_characters.txt")

base_path <- "~/tmp/filenametest"
dir.create(path=base_path)
failed_characters <- c()
overwritten_characters <- c()

for (current_char in good_windows_characters_l) {
  filename <- file.path(base_path, current_char)
  success <- FALSE
  n_files_start <- length(list.files(path=base_path))
  try({
    writeLines(text=current_char, con=filename)
    success <- TRUE
  })
  if (!success) {
    message(current_char, " failed")
    failed_characters <- c(failed_characters, current_char)
  } else {
    n_files <- length(list.files(path=base_path)) - n_files_start
    if (n_files != 1) {
      message(current_char, " overwrote")
      overwritten_characters <- c(overwritten_characters, current_char)
    }    
  }
}