PHP: server seems to continue working with old contents of files

[kennaar]

We are having a rather annoying issue. We're not sure if mutagen is the problem but we never experienced this before when we used a virtualbox share.

We are using mutagen to sync PHP code to a virtualbox machine.
Sometimes it seems like the server keeps working with old contents. E.g.: a crash happens on code that doesn't exist anymore. When we inspect the file causing the crash (or any other file) it is identical to what we have locally so the file did get synced. Reproducing the problem is very difficult because it happens very randomly.

Our workaround:

• Opening the file on the server with nano or vim and re-saving
• Restarting apache

This is a very annoying problem and we have lost quite some time debugging crashes that magically got fixed by doing one of the above options.

Is there anything we can do to actually debug this problem and find the cause?
Thanks in advance.

[aurora]

Do you have an opcode-cache or something like this enabled in the php installation on your vm? I am using mutagen to sync php code to a dev-server, too and never experienced such problem.

[xenoscopic]

@knnnrd Sorry to hear that! Can you tell me how the PHP code is being reloaded on the server? Is it some sort of hot-reload plugin? I'm not that familiar with the PHP ecosystem. Maybe the fact that Mutagen is swapping in files with a rename is causing issues? I'd be surprised if that were the case, but anything is possible. If you can tell me a little bit about your setup, I can have a look at the plugins you're using and see if there are any obvious conflicts/solutions. If I can reproduce, it should be really easy to debug.

[kennaar]

It does seem like the opcode cache is to blame. Clearing it with the following snippet does the trick:

<?php
$result = opcache_reset();
echo $result ? 'Cache reset' : 'Something went wrong';

Clearing it like this is what happens when restarting/loading apache as I understand it.

@havoc-io
A PHP application doesn't have to be "reloaded" in anyway. You can edit and save a file and the changes are reflected the next time you reload the webpage or execute it via the CLI. This is because the webserver interprets the PHP files for every request. However there are some caching mechanisms to improve performance. The most important one is called "OPCache". More information to be found here: https://www.php.net/manual/en/intro.opcache.php

Our setup:
Debian 10
Apache 2.4.38 using mod_php (apache uses PHP as a module to interpret PHP files)
PHP 7.3.4-2
Opcode caching enabled

Thanks!

[xenoscopic]

@knnnrd Thanks for the update, and thanks to @aurora for the opcode cache recommendation.

I'll close this out, but let me know if there's anything else I've missed.

[Toilal]

@knnnrd If you are using composer, make sure you DO NOT use some production flags on composer install like --optimize-autoloader or --classmap-authoritative. Please give us more details on your setup, alpha side and beta side, and details on both filesystem used.

@havoc-io OPCache relies on filesystem timestamps to quickly check that file has changed. Does mutagen properly updates filesystem timestamps on updates ? creation/deletion ? What's behavior when deleting a file and creating the same file again ?

I know that Windows filesystem keeps the same inode ID when deleting and creating the same file again in a short time interval, maybe there's some kind of obscure behavior in one of the involved filesystem that cause this issue.

[kennaar]

Mutagen config:

sync:
  defaults:
    mode: "two-way-resolved"
    ignore:
      vcs: true
      paths:
        - ".DS_Store"
        - "._*"
        - "*~"
        - "*.sw[a-p]"
        - ".idea"

Alpha:
OS: Ubuntu 18.04.3
Filesystem: ext4

Beta:
OS: Debian GNU/Linux buster/sid
Filesystem: ext4

If you are using composer

I'm not sure if that's relevant here since clearing opcache fixes the issue and composer has nothing to do with the opcache. Nevertheless I will try re-dumping the autoloader next time I have the issue.

It seems like I get the issue somewhat often when I switch branches with git if that's even helpful information.

[Toilal]

It may have a relation, see https://getcomposer.org/doc/articles/autoloader-optimization.md

On PHP 5.6+, the class map is also cached in opcache which improves the initialization time greatly. If you make sure opcache is enabled, then the class map should load almost instantly and then class loading is fast.

[kennaar]

It may have a relation, see https://getcomposer.org/doc/articles/autoloader-optimization.md

On PHP 5.6+, the class map is also cached in opcache which improves the initialization time greatly. If you make sure opcache is enabled, then the class map should load almost instantly and then class loading is fast.

Ok that does make sense actually. Thanks. Will try when I have the issue again.

[kennaar]

@Toilal @havoc-io FYI: I was able to reproduce the problem again.
Running composer dump-autoload did nothing. After that I cleared the opcache manually by browsing to the following file with a web browser and it fixed the problem.

<?php
echo (string)opcache_reset();

I had just switched branches with git. A different "pretty" error page package was required in one branch so the visual difference was very obvious.

I will try tweaking my opcache settings and report back.

[xenoscopic]

Hey @knnnrd, sorry for the delayed reply, but was there any resolution to the opcache investigation? I'd really like to figure out what's causing the issue for you. I'm thinking it might be related to #163 (and thus maybe solved by #158). If there's a simple container-based setup that can reproduce the problem, it should be fairly straightforward to debug.