Skip to content

mysql listener: idle mysql database causes rogue listener #98

@burleyb

Description

@burleyb

Description:
While working on a dev database with no changes happening in the db, the listener checkpoint gets stuck on a binlog file that doesn't exist anymore. This causes a rogue that makes it hard to develop because you have to reset the checkpoint every hour or so to restart the listener.

How to reproduce:

  1. If I do show master logs; I get this:
    Log_name File_size
    mysql-bin-changelog.238847 578
    mysql-bin-changelog.238848 578
    mysql-bin-changelog.238849 578
    mysql-bin-changelog.238850 521

  2. I set the checkpoint to start at the earliest log position that exists. mysql-bin-changelog.238847::0. Then restart the bot.

  3. If I make a change to a table that is being listened to, the listener works as expected and catches up to the last log location. The checkpoint points to a location in the last file mysql-bin-changelog.238850

  4. After an amount of time, the listener goes into rogue status with the error.
    Error: ER_MASTER_FATAL_ERROR_READING_BINLOG: Could not find first log file name in binary log index file at Binlog.Sequence._packetToError (/var/task/index.js:84094:14) at Binlog.Sequence.ErrorPacket (/var/task/index.js:84143:17) at Protocol._parsePacket (/var/task/index.js:79237:23) at Parser.write (/var/task/index.js:78556:12) at Protocol.write (/var/task/index.js:78997:16) at Socket. (/var/task/index.js:76656:28) at emitOne (events.js:116:13) at Socket.emit (events.js:211:7) at addChunk (_stream_readable.js:263:12) at readableAddChunk (_stream_readable.js:250:11) -------------------- at Protocol._enqueue (/var/task/index.js:79103:48) at Immediate._start [as _onImmediate] (/var/task/index.js:20906:29) at runCallback (timers.js:794:20) at tryOnImmediate (timers.js:752:5) at processImmediate [as _immediateCallback] (timers.js:729:5) code: 'ER_MASTER_FATAL_ERROR_READING_BINLOG', errno: 1236, sqlMessage: 'Could not find first log file name in binary log index file', sqlState: 'HY000'

  5. Doing show master logs; indicates that the checkpoint location points to a file that has already been rotated out and deleted.

Expected behavior:
Our install is mysql setup on RDS with a standard configuration. It rotates files pretty quickly, even though there is no activity. I realize that I can change mysql parameters to not rotate and delete as quickly, and that this wouldn't be an issue on production. But I would expect the listener not to get stuck regardless of inactivity... Among other issues, it wastes a lot of time during development to have to reset the checkpoint.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions