500 internal server

[schmidtytg]

Mattermost Version: 5.28.1
Database Schema Version: 5.28.0
Database: mysql
Mysql 8.0.22
Zoom plugin: 1.5

I'm able to resolve the issue by disabling the plugin and then enabling it again. While this usually isn't a big deal, I'm not always available to reset the plugin.

I tried completely uninstalling and reinstalling the plugin a couple months ago, but the same behavior seems to be continuing.

The issue seems to have started around when I upgraded mysql from 5.7 to 8.0. Not seeing anything in the mysql logs to indicate an issue with mysql though. The mattermost.log doesn't give me much to go off of either.

From what I can tell there's nothing logged until the plugin seemingly stops working.

{"level":"error","ts":1613014032.4081192,"caller":"mlog/log.go:229","msg":"Plugin failed to ServeHTTP, RPC call failed","plugin_id":"zoom","error":"connection is shut down"}
{"level":"error","ts":1613014037.2250252,"caller":"http/h2_bundle.go:4218","msg":"http2: server connection error from 24.220.147.23:1628: connection error: PROTOCOL_ERROR","source":"httpserver"}
{"level":"error","ts":1613014037.4109783,"caller":"mlog/log.go:229","msg":"Plugin failed to ServeHTTP, muxBroker couldn't Accept request body connection","plugin_id":"zoom","error":"timeout waiting for accept"}
{"level":"error","ts":1613014037.4110491,"caller":"mlog/log.go:229","msg":"Plugin failed to ServeHTTP, muxBroker couldn't accept connection","plugin_id":"zoom","serve_http_stream_id":246,"error":"timeout waiting for accept"}

Any suggestions?

PS. This is my first bug report on github. Please let me know if there's anything I can do better in the future when reporting issues.

[schmidtytg]

I forgot to add that OAuth is not enabled.

[aaronrothschild]

Hey @schmidtytg Thanks for filing this (and welcome to Github)!

We'll review it in our team meeting on Monday/Tuesday to triage it and see if we need more info from you, etc.

[aaronrothschild]

I forgot to add that OAuth is not enabled.

Sorry, could you elaborate? are you using JWT or OAuth for your Zoom app authentication method?

[schmidtytg]

We're using JWT, not OAuth.

[schmidtytg]

After doing an upgrade yesterday to server version 5.31.1, all hell broke loose. It turns out I hadn't updated the character set on all the tables after updating mysql to 8.0.

I've updated the collation of all the database tables to utf8mb4_unicode_ci. Things have been looking much better performance-wise since.

Given this major revelation/rtfm, it would seem prudent to wait and see if the issue with the zoom plug-in has been resolved as well.

[schmidtytg]

Unfortunately this issue just reared its ugly head again.

{"level":"error","ts":1614354624.8175862,"caller":"mlog/log.go:232","msg":"Plugin failed to ServeHTTP, RPC call failed","plugin_id":"zoom","error":"connection is shut down"}
{"level":"error","ts":1614354627.7682946,"caller":"http/h2_bundle.go:4218","msg":"http2: server connection error from 69.67.17.128:4721: connection error: PROTOCOL_ERROR","source":"httpserver"}
{"level":"error","ts":1614354629.8191347,"caller":"mlog/log.go:232","msg":"Plugin failed to ServeHTTP, muxBroker couldn't accept connection","plugin_id":"zoom","serve_http_stream_id":346,"error":"timeout waiting for accept"}
{"level":"error","ts":1614354629.8191993,"caller":"mlog/log.go:232","msg":"Plugin failed to ServeHTTP, muxBroker couldn't Accept request body connection","plugin_id":"zoom","error":"timeout waiting for accept"}
{"level":"error","ts":1614354629.8880343,"caller":"mlog/log.go:232","msg":"Plugin failed to ServeHTTP, RPC call failed","plugin_id":"zoom","error":"connection is shut down"}

I'm not sure if this is relevant or not, but when I first set Enable Plugin to False and hit save, the radio button went back to True. When I set it False and saved a second time, False stayed selected. The plugin started working again after setting it to True again. Same symptoms with users getting the 500 internal server error again.

[schmidtytg]

I just had another issue with the plugin crashing. This sent the CPU of my mattermost server up to 100% for about 10 minutes before I was able to disable/enabled the Zoom plugin. Once the zoom plugin was disabled and then reenabled, the CPU dropped back to to its normal levels.

This was the process that was running at high CPU: plugin-linux-am
There were two of these processes running at the same time. Both were the highest CPU processes at the time.

[schmidtytg]

This continues to happen. Any help would be appreciated.

[hanzei]

Could you please set the log level of your MM server to Debug and check if you see any related messages around the crash?