Possible regression - there is no tailed log from a failed unit #531
Description
When #518 was merged, I'm pretty sure we were able to see the unit log from a failed unit (zaza.model.run_on_unit and tail).
However, I've realized that there is no tailed log from recent CI jobs. For example, the following one has [ERROR] Juju log for octavia/2 as expected but there is no following lines of the actual log. But we see trace back from zaza instead.
2022-08-10 01:47:53.195019 | focal-medium | 2022-08-10 01:47:53 [INFO] Deploy of bundle completed.
2022-08-10 01:47:53.707547 | focal-medium | 2022-08-10 01:47:53 [INFO] Waiting for environment to settle
2022-08-10 01:47:53.708863 | focal-medium | 2022-08-10 01:47:53 [INFO] Timeout for deployment to settle set to: 7200
2022-08-10 01:47:53.709183 | focal-medium | 2022-08-10 01:47:53 [INFO] Maximum resolve count for deployment set to: 3
2022-08-10 01:47:53.710657 | focal-medium | 2022-08-10 01:47:53 [INFO] Waiting for application states to reach targeted states.
2022-08-10 01:47:53.789756 | focal-medium | 2022-08-10 01:47:53 [WARNING] unknown facade EnvironUpgrader
2022-08-10 01:47:53.789871 | focal-medium | 2022-08-10 01:47:53 [WARNING] unexpected facade EnvironUpgrader found, unable to decipher version to use
2022-08-10 01:47:54.363949 | focal-medium | 2022-08-10 01:47:54 [INFO] Waiting for an application to be present
2022-08-10 01:47:54.364242 | focal-medium | 2022-08-10 01:47:54 [INFO] Now checking workload status and status messages
2022-08-10 01:51:45.821781 | focal-medium | 2022-08-10 01:51:45 [INFO] Application rabbitmq-server is ready.
2022-08-10 01:52:59.849512 | focal-medium | 2022-08-10 01:52:59 [WARNING] Unit octavia/2 is in error state. Attempt number 1 to resolve
2022-08-10 01:52:59.850338 | focal-medium | 2022-08-10 01:52:59 [INFO] Resolving unit: octavia/2
2022-08-10 01:54:37.322669 | focal-medium | 2022-08-10 01:54:37 [INFO] Application mysql-innodb-cluster is ready.
2022-08-10 01:55:07.064651 | focal-medium | 2022-08-10 01:55:07 [INFO] Application glance-simplestreams-sync is ready.
2022-08-10 01:55:08.769919 | focal-medium | 2022-08-10 01:55:08 [WARNING] Unit octavia/2 is in error state. Attempt number 2 to resolve
2022-08-10 01:55:08.771078 | focal-medium | 2022-08-10 01:55:08 [INFO] Resolving unit: octavia/2
2022-08-10 01:55:34.084292 | focal-medium | 2022-08-10 01:55:34 [WARNING] Unit octavia/2 is in error state. Attempt number 3 to resolve
2022-08-10 01:55:34.085231 | focal-medium | 2022-08-10 01:55:34 [INFO] Resolving unit: octavia/2
2022-08-10 01:55:42.918623 | focal-medium | 2022-08-10 01:55:42 [WARNING] Unit octavia/1 is in error state. Attempt number 1 to resolve
2022-08-10 01:55:42.919040 | focal-medium | 2022-08-10 01:55:42 [INFO] Resolving unit: octavia/1
2022-08-10 01:57:15.165771 | focal-medium | 2022-08-10 01:57:15 [INFO] Application keystone-mysql-router is ready.
2022-08-10 01:57:31.165681 | focal-medium | 2022-08-10 01:57:31 [INFO] Application neutron-mysql-router is ready.
2022-08-10 01:57:59.779868 | focal-medium | 2022-08-10 01:57:59 [WARNING] Unit octavia/0 is in error state. Attempt number 1 to resolve
2022-08-10 01:57:59.781388 | focal-medium | 2022-08-10 01:57:59 [INFO] Resolving unit: octavia/0
2022-08-10 01:58:16.313053 | focal-medium | 2022-08-10 01:58:16 [INFO] Application placement-mysql-router is ready.
2022-08-10 01:58:18.376355 | focal-medium | 2022-08-10 01:58:18 [INFO] Application glance-mysql-router is ready.
2022-08-10 01:58:24.951056 | focal-medium | 2022-08-10 01:58:24 [INFO] Application nova-mysql-router is ready.
2022-08-10 01:58:42.649698 | focal-medium | 2022-08-10 01:58:42 [ERROR] {'default_alias': 'zaza-dd75b148e4fc'}
2022-08-10 01:58:42.649809 | focal-medium | 2022-08-10 01:58:42 [ERROR] Model default_alias (zaza-dd75b148e4fc)
2022-08-10 01:58:42.942621 | focal-medium | 2022-08-10 01:58:42 [ERROR] Applications in error state: octavia
2022-08-10 01:58:42.942687 | focal-medium | 2022-08-10 01:58:42 [ERROR] Units in error state: octavia/2
2022-08-10 01:58:42.942699 | focal-medium | 2022-08-10 01:58:42 [ERROR] Juju log for octavia/2
2022-08-10 01:58:44.532324 | focal-medium | Traceback (most recent call last):
2022-08-10 01:58:44.532422 | focal-medium | File "/home/ubuntu/src/review.opendev.org/openstack/charm-octavia/src/.tox/func-target/lib/python3.8/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 135, in run_env_deployment
2022-08-10 01:58:44.532462 | focal-medium | deploy.deploy(
2022-08-10 01:58:44.532483 | focal-medium | File "/home/ubuntu/src/review.opendev.org/openstack/charm-octavia/src/.tox/func-target/lib/python3.8/site-packages/zaza/charm_lifecycle/deploy.py", line 404, in deploy
2022-08-10 01:58:44.532498 | focal-medium | zaza.model.wait_for_application_states(
2022-08-10 01:58:44.532571 | focal-medium | File "/home/ubuntu/src/review.opendev.org/openstack/charm-octavia/src/.tox/func-target/lib/python3.8/site-packages/zaza/__init__.py", line 221, in _wrapper
2022-08-10 01:58:44.532597 | focal-medium | return future.result(timeout)
2022-08-10 01:58:44.532609 | focal-medium | File "/usr/lib/python3.8/concurrent/futures/_base.py", line 444, in result
2022-08-10 01:58:44.532621 | focal-medium | return self.__get_result()
2022-08-10 01:58:44.532632 | focal-medium | File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
2022-08-10 01:58:44.532644 | focal-medium | raise self._exception
2022-08-10 01:58:44.532655 | focal-medium | File "/home/ubuntu/src/review.opendev.org/openstack/charm-octavia/src/.tox/func-target/lib/python3.8/site-packages/zaza/__init__.py", line 206, in _runner
2022-08-10 01:58:44.532666 | focal-medium | return await f(*args, **kwargs)
2022-08-10 01:58:44.532699 | focal-medium | File "/home/ubuntu/src/review.opendev.org/openstack/charm-octavia/src/.tox/func-target/lib/python3.8/site-packages/zaza/model.py", line 1635, in async_wait_for_application_states
2022-08-10 01:58:44.532712 | focal-medium | ok = check_unit_workload_status(
2022-08-10 01:58:44.532724 | focal-medium | File "/home/ubuntu/src/review.opendev.org/openstack/charm-octavia/src/.tox/func-target/lib/python3.8/site-packages/zaza/model.py", line 1360, in check_unit_workload_status
2022-08-10 01:58:44.532736 | focal-medium | check_model_for_hard_errors(model)
2022-08-10 01:58:44.532761 | focal-medium | File "/home/ubuntu/src/review.opendev.org/openstack/charm-octavia/src/.tox/func-target/lib/python3.8/site-packages/zaza/model.py", line 1338, in check_model_for_hard_errors
2022-08-10 01:58:44.532773 | focal-medium | raise UnitError(errored_units)
2022-08-10 01:58:44.532784 | focal-medium | zaza.model.UnitError: Units octavia/2 in error state
2022-08-10 01:58:44.532803 | focal-medium |
2022-08-10 01:58:44.532816 | focal-medium | During handling of the above exception, another exception occurred:
2022-08-10 01:58:44.532827 | focal-medium |
2022-08-10 01:58:44.532838 | focal-medium | Traceback (most recent call last):
2022-08-10 01:58:44.532852 | focal-medium | File "/home/ubuntu/src/review.opendev.org/openstack/charm-octavia/src/.tox/func-target/bin/functest-run-suite", line 8, in <module>
2022-08-10 01:58:44.532876 | focal-medium | sys.exit(main())
2022-08-10 01:58:44.532905 | focal-medium | File "/home/ubuntu/src/review.opendev.org/openstack/charm-octavia/src/.tox/func-target/lib/python3.8/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 383, in main
2022-08-10 01:58:44.532928 | focal-medium | func_test_runner(
2022-08-10 01:58:44.532952 | focal-medium | File "/home/ubuntu/src/review.opendev.org/openstack/charm-octavia/src/.tox/func-target/lib/python3.8/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 295, in func_test_runner
2022-08-10 01:58:44.532978 | focal-medium | run_env_deployment(env_deployment, keep_model=preserve_model,
2022-08-10 01:58:44.533002 | focal-medium | File "/home/ubuntu/src/review.opendev.org/openstack/charm-octavia/src/.tox/func-target/lib/python3.8/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 182, in run_env_deployment
2022-08-10 01:58:44.533090 | focal-medium | failure_report(model_aliases)
2022-08-10 01:58:44.533116 | focal-medium | File "/home/ubuntu/src/review.opendev.org/openstack/charm-octavia/src/.tox/func-target/lib/python3.8/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 82, in failure_report
2022-08-10 01:58:44.533127 | focal-medium | log_output = zaza.model.run_on_unit(
2022-08-10 01:58:44.533139 | focal-medium | KeyError: 'stdout'
2022-08-10 01:58:44.717953 | focal-medium | 2022-08-10 01:58:44 [ERROR] Fatal error on SSL transport
2022-08-10 01:58:44.718024 | focal-medium | protocol: <asyncio.sslproto.SSLProtocol object at 0x7fd3e0aef8b0>
2022-08-10 01:58:44.718035 | focal-medium | transport: <_SelectorSocketTransport closing fd=7>
2022-08-10 01:58:44.718042 | focal-medium | Traceback (most recent call last):
2022-08-10 01:58:44.718049 | focal-medium | File "/usr/lib/python3.8/asyncio/selector_events.py", line 910, in write
2022-08-10 01:58:44.718057 | focal-medium | n = self._sock.send(data)
2022-08-10 01:58:44.718064 | focal-medium | OSError: [Errno 9] Bad file descriptor
2022-08-10 01:58:44.718073 | focal-medium |
2022-08-10 01:58:44.718113 | focal-medium | During handling of the above exception, another exception occurred:
2022-08-10 01:58:44.718129 | focal-medium |
2022-08-10 01:58:44.718137 | focal-medium | Traceback (most recent call last):
2022-08-10 01:58:44.718176 | focal-medium | File "/usr/lib/python3.8/asyncio/sslproto.py", line 685, in _process_write_backlog
2022-08-10 01:58:44.718186 | focal-medium | self._transport.write(chunk)
2022-08-10 01:58:44.718193 | focal-medium | File "/usr/lib/python3.8/asyncio/selector_events.py", line 916, in write
2022-08-10 01:58:44.718200 | focal-medium | self._fatal_error(exc, 'Fatal write error on socket transport')
2022-08-10 01:58:44.718206 | focal-medium | File "/usr/lib/python3.8/asyncio/selector_events.py", line 711, in _fatal_error
2022-08-10 01:58:44.718213 | focal-medium | self._force_close(exc)
2022-08-10 01:58:44.718220 | focal-medium | File "/usr/lib/python3.8/asyncio/selector_events.py", line 723, in _force_close
2022-08-10 01:58:44.718226 | focal-medium | self._loop.call_soon(self._call_connection_lost, exc)
2022-08-10 01:58:44.718233 | focal-medium | File "/usr/lib/python3.8/asyncio/base_events.py", line 719, in call_soon
2022-08-10 01:58:44.718240 | focal-medium | self._check_closed()
2022-08-10 01:58:44.718247 | focal-medium | File "/usr/lib/python3.8/asyncio/base_events.py", line 508, in _check_closed
2022-08-10 01:58:44.718253 | focal-medium | raise RuntimeError('Event loop is closed')
2022-08-10 01:58:44.718260 | focal-medium | RuntimeError: Event loop is closed
2022-08-10 01:58:44.718586 | focal-medium | 2022-08-10 01:58:44 [ERROR] Task was destroyed but it is pending!
2022-08-10 01:58:44.718609 | focal-medium | task: <Task pending name='Task-137' coro=<WebSocketCommonProtocol.transfer_data() running at /home/ubuntu/src/review.opendev.org/openstack/charm-octavia/src/.tox/func-target/lib/python3.8/site-packages/websockets/protocol.py:674> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7fd3e0adc100>()]> cb=[<TaskWakeupMethWrapper object at 0x7fd3e0afa160>(), _wait.<locals>._on_completion() at /usr/lib/python3.8/asyncio/tasks.py:518]>
2022-08-10 01:58:44.718843 | focal-medium | 2022-08-10 01:58:44 [ERROR] Task was destroyed but it is pending!
2022-08-10 01:58:44.718867 | focal-medium | task: <Task pending name='Task-139' coro=<WebSocketCommonProtocol.close_connection() running at /home/ubuntu/src/review.opendev.org/openstack/charm-octavia/src/.tox/func-target/lib/python3.8/site-packages/websockets/protocol.py:1019> wait_for=<Task pending name='Task-137' coro=<WebSocketCommonProtocol.transfer_data() running at /home/ubuntu/src/review.opendev.org/openstack/charm-octavia/src/.tox/func-target/lib/python3.8/site-packages/websockets/protocol.py:674> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7fd3e0adc100>()]> cb=[<TaskWakeupMethWrapper object at 0x7fd3e0afa160>(), _wait.<locals>._on_completion() at /usr/lib/python3.8/asyncio/tasks.py:518]>>
2022-08-10 01:58:44.720619 | focal-medium | Exception ignored in: <generator object WebSocketCommonProtocol.close_connection at 0x7fd3e0af0ba0>
2022-08-10 01:58:44.720646 | focal-medium | Traceback (most recent call last):
2022-08-10 01:58:44.720656 | focal-medium | File "/home/ubuntu/src/review.opendev.org/openstack/charm-octavia/src/.tox/func-target/lib/python3.8/site-packages/websockets/protocol.py", line 1056, in close_connection
2022-08-10 01:58:44.720664 | focal-medium | if (yield from self.wait_for_connection_lost()):
2022-08-10 01:58:44.720671 | focal-medium | File "/home/ubuntu/src/review.opendev.org/openstack/charm-octavia/src/.tox/func-target/lib/python3.8/site-packages/websockets/protocol.py", line 1077, in wait_for_connection_lost
2022-08-10 01:58:44.720678 | focal-medium | yield from asyncio.wait_for(
2022-08-10 01:58:44.720685 | focal-medium | File "/usr/lib/python3.8/asyncio/tasks.py", line 472, in wait_for
2022-08-10 01:58:44.720692 | focal-medium | timeout_handle = loop.call_later(timeout, _release_waiter, waiter)
2022-08-10 01:58:44.720699 | focal-medium | File "/usr/lib/python3.8/asyncio/base_events.py", line 687, in call_later
2022-08-10 01:58:44.720705 | focal-medium | timer = self.call_at(self.time() + delay, callback, *args,
2022-08-10 01:58:44.720712 | focal-medium | File "/usr/lib/python3.8/asyncio/base_events.py", line 698, in call_at
2022-08-10 01:58:44.720719 | focal-medium | self._check_closed()
2022-08-10 01:58:44.720725 | focal-medium | File "/usr/lib/python3.8/asyncio/base_events.py", line 508, in _check_closed
2022-08-10 01:58:44.720732 | focal-medium | raise RuntimeError('Event loop is closed')
2022-08-10 01:58:44.720738 | focal-medium | RuntimeError: Event loop is closed
2022-08-10 01:58:44.720938 | focal-medium | 2022-08-10 01:58:44 [ERROR] Task was destroyed but it is pending!
2022-08-10 01:58:44.720962 | focal-medium | task: <Task pending name='Task-138' coro=<WebSocketCommonProtocol.keepalive_ping() running at /home/ubuntu/src/review.opendev.org/openstack/charm-octavia/src/.tox/func-target/lib/python3.8/site-packages/websockets/protocol.py:977> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7fd3e00b8fa0>()]>>
2022-08-10 01:58:44.721201 | focal-medium | 2022-08-10 01:58:44 [ERROR] Task was destroyed but it is pending!
2022-08-10 01:58:44.721225 | focal-medium | task: <Task pending name='Task-145' coro=<Connection._pinger() running at /home/ubuntu/src/review.opendev.org/openstack/charm-octavia/src/.tox/func-target/lib/python3.8/site-packages/juju/client/connection.py:578> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7fd3e0afaa90>()]>>
2022-08-10 01:58:44.721381 | focal-medium | 2022-08-10 01:58:44 [ERROR] Task was destroyed but it is pending!
2022-08-10 01:58:44.721400 | focal-medium | task: <Task pending name='Task-140' coro=<Connection._receiver() running at /home/ubuntu/src/review.opendev.org/openstack/charm-octavia/src/.tox/func-target/lib/python3.8/site-packages/juju/client/connection.py:530> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7fd3e0af3490>()]>>
2022-08-10 01:58:44.721688 | focal-medium | 2022-08-10 01:58:44 [ERROR] Task was destroyed but it is pending!
2022-08-10 01:58:44.721711 | focal-medium | task: <Task pending name='Task-4330' coro=<WebSocketCommonProtocol.recv() running at /home/ubuntu/src/review.opendev.org/openstack/charm-octavia/src/.tox/func-target/lib/python3.8/site-packages/websockets/protocol.py:416> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7fd3e00b32b0>()]> cb=[create_task_with_handler.<locals>._task_result_exp_handler(task_name='tmp', logger=<Logger juju....ection (INFO)>)() at /home/ubuntu/src/review.opendev.org/openstack/charm-octavia/src/.tox/func-target/lib/python3.8/site-packages/juju/jasyncio.py:64]>
2022-08-10 01:58:44.721873 | focal-medium | 2022-08-10 01:58:44 [ERROR] Task was destroyed but it is pending!
2022-08-10 01:58:44.721893 | focal-medium | task: <Task pending name='Task-4331' coro=<Event.wait() running at /usr/lib/python3.8/asyncio/locks.py:309> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7fd3e1bfdd60>()]>>
2022-08-10 01:58:44.958119 | focal-medium | ERROR: InvocationError for command /home/ubuntu/src/review.opendev.org/openstack/charm-octavia/src/.tox/func-target/bin/functest-run-suite --keep-model --bundle focal-xena-ha (exited with code 1)
2022-08-10 01:58:44.961271 | focal-medium | ___________________________________ summary ____________________________________
2022-08-10 01:58:44.961319 | focal-medium | ERROR: func-target: commands failed
2022-08-10 01:58:45.057241 | focal-medium | ERROR: InvocationError for command /home/ubuntu/src/review.opendev.org/openstack/charm-octavia/.tox/func-target/bin/tox -e func-target focal-xena-ha (exited with code 1)
2022-08-10 01:58:45.061239 | focal-medium | ___________________________________ summary ____________________________________
2022-08-10 01:58:45.061328 | focal-medium | ERROR: func-target: commands failed
2022-08-10 01:58:45.412805 | focal-medium | ERROR
2022-08-10 01:58:45.413857 | focal-medium | {
2022-08-10 01:58:45.414049 | focal-medium | "delta": "0:13:22.343763",
2022-08-10 01:58:45.414328 | focal-medium | "end": "2022-08-10 01:58:45.156298",
2022-08-10 01:58:45.414453 | focal-medium | "msg": "non-zero return code",
2022-08-10 01:58:45.414537 | focal-medium | "rc": 1,
2022-08-10 01:58:45.414614 | focal-medium | "start": "2022-08-10 01:45:22.812535"
2022-08-10 01:58:45.414687 | focal-medium | }