Losing connection to database, not recovering #27
Description
Hi
Having fluentd running as DaemonSet in kubernetes based on
fluentd:v1.2.6-debian using the current 1.0.0 fluent-plugin-pgjson plugin version.
After some time we lose connection to the database and then miss some logs in the db.
2018-10-26 06:59:21 +0000 [warn]: #0 fluent/log.rb:342:warn: failed to flush the buffer. retry_time=16 next_retry_seconds=2018-10-26 06:59:21 +0000 chunk="5790cd573c914755aed4e344e7b110b9" error_class=PG::UnableToSend error="no connection to the server\n"
2018-10-26 23:09:19 +0000 [warn]: #0 fluent/log.rb:342:warn: failed to flush the buffer. retry_time=17 next_retry_seconds=2018-10-26 23:09:19 +0000 chunk="5790cd573c914755aed4e344e7b110b9" error_class=PG::UnableToSend error="no connection to the server\n"
2018-10-28 08:00:18 +0000 [warn]: #0 fluent/log.rb:342:warn: failed to flush the buffer. retry_time=18 next_retry_seconds=2018-10-28 08:00:18 +0000 chunk="5790cd573c914755aed4e344e7b110b9" error_class=PG::UnableToSend error="no connection to the server\n"
2018-10-28 12:36:12 +0000 [error]: #0 fluent/log.rb:362:error: failed to flush the buffer, and hit limit for retries. dropping all chunks in the buffer queue. retry_times=19 records=46 error_class=PG::UnableToSend error="no connection to the server\n"
Not being a ruby programmer myself, I'm wondering how to recover from that, creating a new connection to the db somehow, add some retry logic somewhere ?!
Additional Info:
- We split the log stream ("@type copy") and one part goes to graylog, that part is working fine. We don't lose any loglines. It all ends up in Graylog.
- Before, we were using fluentd 0.12 with pg gem 0.21.0. With this setup we reproducibly lost connection to the db after some hours. Then I upgraded fluentd to 1.2.6 to be able to use the latest version of the plugin and a more current version of the pg gem.
Any hints or help appreciated.
Regards