Workbench can't upload files: Unexpected response from server

[smokedlinq]

Trying to upload files to a session and received Unexpected response from server. On the session pod the following error show up:

2022-11-21T21:59:13.566313Z [rsession-Username] ERROR system error 1 (Operation not permitted) [path: /home/Username/.local/share/rstudio/rstudio-server.json]; OCCURRED AT rstudio::core::Error rstudio::core::{anonymous}::changeFileModeImpl(const string&, mode_t) src/cpp/shared_core/FilePath.cpp:314; LOGGED FROM: virtual rstudio::core::Error rstudio::session::prefs::UserStateLayer::writePrefs(const rstudio::core::json::Object&) src/cpp/session/prefs/UserStateLayer.cpp:74

I am able to create folders/files in workbench without issue, it is the upload that is failing.

[colearendt]

Interesting! Sorry to hear about the trouble here!

Can I confirm - are you running the latest version of the product? 2022.07.2? We have had some issues with the upload that we fixed a while back, and want to confirm you are not hitting those. I just confirmed that it works on our demo environment using the helm charts and running 2022.07.2, so we definitely expect this to function 😄

Am I remembering correctly that you are using the EFS CSI as well? We have seen some issues with the EFS CSI as of late, so I am curious if that may be a factor here as well.

[smokedlinq]

@colearendt using the latest chart (0.5.22) and the file.csi.azure.com provisioner, here is the storage class spec:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: rstudio-workbench
provisioner: file.csi.azure.com
allowVolumeExpansion: true
reclaimPolicy: Delete
volumeBindingMode: Immediate
mountOptions:
- mfsymlinks
- actimeo=30

[colearendt]

Very interesting. We need to do more testing with Azure Files CSI. I am suspicious that something is going awry here on the filesystem - have you tried with the NFS-based Azure mounts?

The "Operation not permitted" is coming from an attempt to change the file mode (chmod) to 600, I believe. My suspicion is that something about the filesystem is not allowing this operation (i.e. root squash, uid restrictions, etc.). You could test this fairly easily in a container by starting a shell as root, (i.e. kubectl exec -it container-name -- bash), creating a file (touch my-file) and then trying to chmod it (chmod 600 my-file).

Would you mind giving that a shot and letting us know how it goes?

[smokedlinq]

chmod 600 my-file doesn't change the permissions, stays as 777. I did try setting the uid/gid to 999 and that all looks fine, but it isn't changing any permissions for the home directory to the user (not sure if it should or can).

[MariaSemple]

Hi again @smokedlinq, just updating here. Similarly to #327, we have planned work to create an Azure AKS Reference Architecture. That will have more recommendations for the best way to configure Workbench in Azure AKS, including a recommendation to use Azure NetApp Files.