Docker startup error.

[Cwavs]

I can't seem to get the docker image to start anymore. The lpgs indicare, the issue seems to be caused by line 271 of docker.cjs, but I'm not a js dev and that file seems minified, so I'm not sure specifically what the issue is, and the server doesn't seem to be getting far enough to log more information.

I've confirmed that at least version 3.9.1 starts correctly so I'll try to do some more testing to isolate which version it stops starting for me.

Any ideas?

[cpvalente]

Hi @Cwavs ,

I dont immediately see any reason for this behaviour.
Ontime Cloud is also built on docker and it is running the latest version without issues

It would be interesting to hear which version is the last that you can run so we can try to isolate a change

At this point, I suspect the issue to be external to docker (either in your docker compose, nodejs version, or some incompatible data in the mapped volume)

If you could provide the exact docker error along with your node version and docker compose I can try to follow up

[Cwavs]

Hi @cpvalente, thanks for the quick response.

I thought it was odd no one else had reported it!

I think I've gotten to the crux of the issue. It looks to be some sort of permissions issue. I was actually confused in the original issue. It wasn't version 3.9.1, it was 3.0.0! Further testing revealed that the issue only began in 3.4.0, coincidentally when the switch from external to data happened. So I re-tested on 3.3.3 with the mount set to the external folder and I'm getting a new error (one I can paste now as well)!

node:internal/fs/utils:350
    throw err;
    ^

Error: EACCES: permission denied, open '/external/app-state.json'
    at Object.openSync (node:fs:603:3)
    at writeFileSync (node:fs:2324:35)
    at qV (/app/server/docker.cjs:303:7383)
    at A8e (/app/server/docker.cjs:303:7560)
    at Object.<anonymous> (/app/server/docker.cjs:303:7583)
    at Module._compile (node:internal/modules/cjs/loader:1256:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1310:10)
    at Module.load (node:internal/modules/cjs/loader:1119:32)
    at Module._load (node:internal/modules/cjs/loader:960:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:86:12) {
  errno: -13,
  syscall: 'open',
  code: 'EACCES',
  path: '/external/app-state.json'
}

Node.js v18.18.2

The weird thing is that this seems to happen regardless of what permissions are actually on the file or folder (I've even tried 777), so I'm a bit lost there. If I remove the mount in question it does seem to let it start on newer versions, which heavily implies that it is something related to that mount. I have no idea what's wrong though.

I have had a quick look and I couldn't find any similar issues. Have you seen something like this before?

The newer versions produce a longer error which I've uploaded as a text file!

out.txt

[cpvalente]

Thank you @Cwavs

Can you provide your docker-compose definition?

It sounds like you are trying to migrate an old setup. It could be simpler to create a new docker compose based on the current docs and map your volumes and networks to it

[Cwavs]

No problem!

I've not had chance to grab it yet, but I'll send my quadlet your way, I'll try and do that tomorrow (I'm using Podman, but I'm fairly certain it's not the issue, as I've never had any problems specific to it before)

I was originally trying to yes, I was moving the container to a new device. However when I started getting issues, the first thing I tried was wiping everything clean and starting again. I remade the quadlet, and pointed the volumes over to a new empty folder. The current quadlet, and the old one were both fairly recent. I think I started using this with the 3.9.x series!

[Cwavs]

Hi @cpvalente

I've uploaded the quadlet as it currently is! It's worth baring in mind this includes changes made for my testing. The original quadlet for example pointed it's mount towards /data and used the latest tag. I've appended .txt to the end of the file so that I could upload it, but it's otherwise untouched.

Let me know if you spot any errors on my part (I wouldn't be surprised if I've been missing something blindingly obvious the whole time)!

ontime.container.txt

[cpvalente]

Hi @Cwavs

I dont see anything wrong, but I am not sure what you mean here

I've uploaded the quadlet as it currently is! It's worth baring in mind this includes changes made for my testing. The original quadlet for example pointed it's mount towards /data and used the latest tag. I've appended .txt to the end of the file so that I could upload it, but it's otherwise untouched.

Are you saying that the version in this ini file works well?

The first thing that I see here is that you are using the docker image v3.3.0 and the old external directory
Could you try the configuration for the latest version? Something like

[Container]
ContainerName=ontime
Environment=TZ=Europe/London
Image=docker.io/getontime/ontime:v3.14.0 # <---------
PublishPort=80:4001
PublishPort=8888:8888/udp
PublishPort=9999:9999/udp
Pull=newer
Volume=/home/ontime/ontime:/data/ # <---------
AutoUpdate=registry

[Install]
WantedBy=default.target

[Cwavs]

Thanks again for the quick response @cpvalente

Sorry if I was unclear there, it's a complicated statement to summarise. That version does not work. It's probably easiest to explain from the start so we're on the same page:

1. I'm migrating the server from another device, so I copy over the volume and Quadlet. However the container fails to start with the error you can see in the out.txt file in this comment Docker startup error. #1528 (comment)
2. Thinking I messed something up with the migration, I move both the original quadlet and volume to another location, and remake them from scratch so that I can compare and find the issue, however the same problem happens.
3. I'm now very confused, so I start trying some older versions of Ontime, wondering if the bug was introduced recently and the container on the old device hadn't updated for some reason.
4. I eventually find that any version prior to 3.4.0 works, with 3.3.3 being the latest version that runs. I then notice that's when the switch from /external to /data is made.
5. I try switching the mount to /external on 3.3.3 to see what happens and that's when the short error in this comment Docker startup error. #1528 (comment) is seen.
6. I then, out of curiosity, swap back to latest whilst keeping the volume the same (/external meaning /data isn't mounted) and Ontime starts correctly.
7. I then switch back to 3.3.0 (I think I meant to do 3.3.3 instead, but made a typo) to make sure I didn't miss anything obvious whilst testing. This is when that quadlet is from.

Again sorry for the lack of clarity there. I'm aware not only is this issue confusing, but the testing that I've done is too. If you want me to clarify further I am willing too.

I've made the modifications you've point out above to the quadlet and I am now getting the same longer error again. In case I've missed a crucial difference here, I've uploaded the journalctl output again.

out.txt

[cpvalente]

Hi @Cwavs , I apologise for the slow replies. I dont have any many clear leads here to continue here

The error seems to be clearly indicating a lack of privilege on an attempted read (as you already know)
This is a normal issue in mapped volumes where the volume is under a root user directory. A google search for podman permission denied or the generic error nodewriteFileSync Error: EACCES can lead you to find some clues. But again, this will mostly be related to your setup

To me, it remains mysterious as why this would break suddenly as you go through versions, I dont see any changes that would provoke a change of requirements on permission levels

Some weak leads:

• you have the incorrect node version and should be using something >=20
• double check the permissions of the user running the container and the write volume
• try and run the app using docker, just to rule out weird issues with podman

Sorry this wasnt particularly helpful
Maybe @jwetzell has some experience that could help here

[Cwavs]

No worries. Thanks for trying! You've been very accommodating which I appreciate.

Any reason why the node version would be wrong? I would have assumed that would be part of the docker image, if not I guess I'm just using whatever comes with Fedora 41? I'll see if I can update that.

Yeah I'm particularly confused as the mount does have... well as open permissions as I can give it, and the podman instance is running as root, so should have full access regardless... I guess I'll give a Docker instance a go as well.

[jwetzell]

I don't have much knowledge of podman.

• The Node version above shouldn't matter as this is running in a container.
• I would either start with whatever version was running before the migration OR the latest version (v3.14.0) and double check what the volume mount is for that version /external vs /data.
• Another check you might have done but I missed is does that file actually exist in the /home/ontime/ontime directory

[Cwavs]

I believe I have found the source of my error now, and I realise why we couldn't diagnose it. It was an external configuration issue. It was selinux. The container was running on Fedora which enables this by default. I guess I must have disabled it when I set up the original machine and forgot. I assumed that the included container policy would have allowed this access, but I'm now wondering if that only provides access to the local users folders (e.g root containers can only access folders in the root home dir, or something like that). Either way I disabled it and ontime started fine! Sorry for all the trouble and thanks for all the help I appreciate it! @cpvalente @jwetzell