Skip to content
This repository was archived by the owner on Jan 27, 2023. It is now read-only.
This repository was archived by the owner on Jan 27, 2023. It is now read-only.

Dead locks inside Anchore #1391

Open
Open
Dead locks inside Anchore#1391
@pbradly

Description

@pbradly
pbradly
opened on Sep 9, 2022

Is this a request for help?:
Yes

Is this a BUG REPORT or a FEATURE REQUEST? (choose one):
Bug Report

Version of Anchore Engine and Anchore CLI if applicable:
anchore-cli, version 0.9.2
anchore-engine, version v1.0.0
Helm Chart, version anchore-engine-1.19.4

What happened:
We are recently been having issues with deadlocks in the policy container and locks not being held inside the Simple Queue container when trying to refresh the lease. This has completely frozen the application as images are now stuck in an analyzing state due to the policy not being able to process them, and other images stuck in a not_analyzed state due to the simple queue not being able to pick them up.

We have so far attempted to following;

  • Deleted the images previously scanned by anchore
  • Restarted pods by killing them and allowing K8s to spin up new ones
  • Uninstall and re-install the application via Helm
  • Force a fresh install
  • Confirm connectivity to the Database (We host a Postgres RDS instance for the database)

So far none of these have resolved the issue. We have also seen issues where the cli is very slow to respond or doesn't at all, along with heartbeat failures inside the catalog.

What did you expect to happen:
Anchore to scan and report back any vulnerabilities found

Any relevant log output from /var/log/anchore:
Api

[service:api] 2022-09-09 17:20:27+0000 [-] Traceback (most recent call last):
[service:api] 2022-09-09 17:20:27+0000 [-]   File "/usr/local/lib64/python3.8/site-packages/sqlalchemy/engine/base.py", line 3212, in _wrap_pool_connect
[service:api] 2022-09-09 17:20:27+0000 [-]     return fn()
[service:api] 2022-09-09 17:20:27+0000 [-]   File "/usr/local/lib64/python3.8/site-packages/sqlalchemy/pool/base.py", line 307, in connect
[service:api] 2022-09-09 17:20:27+0000 [-]     return _ConnectionFairy._checkout(self)
[service:api] 2022-09-09 17:20:27+0000 [-]   File "/usr/local/lib64/python3.8/site-packages/sqlalchemy/pool/base.py", line 767, in _checkout
[service:api] 2022-09-09 17:20:27+0000 [-]     fairy = _ConnectionRecord.checkout(pool)
[service:api] 2022-09-09 17:20:27+0000 [-]   File "/usr/local/lib64/python3.8/site-packages/sqlalchemy/pool/base.py", line 425, in checkout
[service:api] 2022-09-09 17:20:27+0000 [-]     rec = pool._do_get()
[service:api] 2022-09-09 17:20:27+0000 [-]   File "/usr/local/lib64/python3.8/site-packages/sqlalchemy/pool/impl.py", line 146, in _do_get
[service:api] 2022-09-09 17:20:27+0000 [-]     self._dec_overflow()
[service:api] 2022-09-09 17:20:27+0000 [-]   File "/usr/local/lib64/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
[service:api] 2022-09-09 17:20:27+0000 [-]     compat.raise_(
[service:api] 2022-09-09 17:20:27+0000 [-]   File "/usr/local/lib64/python3.8/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
[service:api] 2022-09-09 17:20:27+0000 [-]     raise exception
[service:api] 2022-09-09 17:20:27+0000 [-]   File "/usr/local/lib64/python3.8/site-packages/sqlalchemy/pool/impl.py", line 143, in _do_get
[service:api] 2022-09-09 17:20:27+0000 [-]     return self._create_connection()
[service:api] 2022-09-09 17:20:27+0000 [-]   File "/usr/local/lib64/python3.8/site-packages/sqlalchemy/pool/base.py", line 253, in _create_connection
[service:api] 2022-09-09 17:20:27+0000 [-]     return _ConnectionRecord(self)
[service:api] 2022-09-09 17:20:27+0000 [-]   File "/usr/local/lib64/python3.8/site-packages/sqlalchemy/pool/base.py", line 368, in __init__
[service:api] 2022-09-09 17:20:27+0000 [-]     self.__connect()
[service:api] 2022-09-09 17:20:27+0000 [-]   File "/usr/local/lib64/python3.8/site-packages/sqlalchemy/pool/base.py", line 611, in __connect
[service:api] 2022-09-09 17:20:27+0000 [-]     pool.logger.debug("Error on connect(): %s", e)
[service:api] 2022-09-09 17:20:27+0000 [-]   File "/usr/local/lib64/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
[service:api] 2022-09-09 17:20:27+0000 [-]     compat.raise_(
[service:api] 2022-09-09 17:20:27+0000 [-]   File "/usr/local/lib64/python3.8/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
[service:api] 2022-09-09 17:20:27+0000 [-]     raise exception
[service:api] 2022-09-09 17:20:27+0000 [-]   File "/usr/local/lib64/python3.8/site-packages/sqlalchemy/pool/base.py", line 605, in __connect
[service:api] 2022-09-09 17:20:27+0000 [-]     connection = pool._invoke_creator(self)
[service:api] 2022-09-09 17:20:27+0000 [-]   File "/usr/local/lib64/python3.8/site-packages/sqlalchemy/engine/create.py", line 578, in connect
[service:api] 2022-09-09 17:20:27+0000 [-]     return dialect.connect(*cargs, **cparams)
[service:api] 2022-09-09 17:20:27+0000 [-]   File "/usr/local/lib64/python3.8/site-packages/sqlalchemy/engine/default.py", line 584, in connect
[service:api] 2022-09-09 17:20:27+0000 [-]     return self.dbapi.connect(*cargs, **cparams)
[service:api] 2022-09-09 17:20:27+0000 [-]   File "/usr/lib64/python3.8/site-packages/psycopg2/__init__.py", line 126, in connect
[service:api] 2022-09-09 17:20:27+0000 [-]     conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
[service:api] 2022-09-09 17:20:27+0000 [-] psycopg2.OperationalError: timeout expired
[service:api] 2022-09-09 17:20:27+0000 [-] 
[service:api] 2022-09-09 17:20:27+0000 [-] 
[service:api] 2022-09-09 17:20:27+0000 [-] The above exception was the direct cause of the following exception:
[service:api] 2022-09-09 17:20:27+0000 [-] 
[service:api] 2022-09-09 17:20:27+0000 [-] Traceback (most recent call last):
[service:api] 2022-09-09 17:20:27+0000 [-]   File "/usr/local/lib/python3.8/site-packages/anchore_engine/subsys/auth/stores/basic.py", line 73, in get_authc_info
[service:api] 2022-09-09 17:20:27+0000 [-]     identity, creds = idp.lookup_user(identifier)
[service:api] 2022-09-09 17:20:27+0000 [-]   File "/usr/local/lib/python3.8/site-packages/anchore_engine/apis/authentication.py", line 40, in lookup_user
[service:api] 2022-09-09 17:20:27+0000 [-]     usr = self.mgr.get_user(username)
[service:api] 2022-09-09 17:20:27+0000 [-]   File "/usr/local/lib/python3.8/site-packages/anchore_engine/subsys/identities.py", line 475, in get_user
[service:api] 2022-09-09 17:20:27+0000 [-]     return db_account_users.get(username, session=self.session)
[service:api] 2022-09-09 17:20:27+0000 [-]   File "/usr/local/lib/python3.8/site-packages/anchore_engine/db/db_account_users.py", line 179, in get
[service:api] 2022-09-09 17:20:27+0000 [-]     usr = session.query(AccountUser).filter_by(username=username).one_or_none()
[service:api] 2022-09-09 17:20:27+0000 [-]   File "/usr/local/lib64/python3.8/site-packages/sqlalchemy/orm/query.py", line 2788, in one_or_none
[service:api] 2022-09-09 17:20:27+0000 [-]     return self._iter().one_or_none()
[service:api] 2022-09-09 17:20:27+0000 [-]   File "/usr/local/lib64/python3.8/site-packages/sqlalchemy/orm/query.py", line 2846, in _iter
[service:api] 2022-09-09 17:20:27+0000 [-]     result = self.session.execute(
[service:api] 2022-09-09 17:20:27+0000 [-]   File "/usr/local/lib64/python3.8/site-packages/sqlalchemy/orm/session.py", line 1688, in execute
[service:api] 2022-09-09 17:20:27+0000 [-]     conn = self._connection_for_bind(bind)
[service:api] 2022-09-09 17:20:27+0000 [-]   File "/usr/local/lib64/python3.8/site-packages/sqlalchemy/orm/session.py", line 1529, in _connection_for_bind
[service:api] 2022-09-09 17:20:27+0000 [-]     return self._transaction._connection_for_bind(
[service:api] 2022-09-09 17:20:27+0000 [-]   File "/usr/local/lib64/python3.8/site-packages/sqlalchemy/orm/session.py", line 747, in _connection_for_bind
[service:api] 2022-09-09 17:20:27+0000 [-]     conn = bind.connect()
[service:api] 2022-09-09 17:20:27+0000 [-]   File "/usr/local/lib64/python3.8/site-packages/sqlalchemy/engine/base.py", line 3166, in connect
[service:api] 2022-09-09 17:20:27+0000 [-]     return self._connection_cls(self, close_with_result=close_with_result)
[service:api] 2022-09-09 17:20:27+0000 [-]   File "/usr/local/lib64/python3.8/site-packages/sqlalchemy/engine/base.py", line 96, in __init__
[service:api] 2022-09-09 17:20:27+0000 [-]     else engine.raw_connection()
[service:api] 2022-09-09 17:20:27+0000 [-]   File "/usr/local/lib64/python3.8/site-packages/sqlalchemy/engine/base.py", line 3245, in raw_connection
[service:api] 2022-09-09 17:20:27+0000 [-]     return self._wrap_pool_connect(self.pool.connect, _connection)
[service:api] 2022-09-09 17:20:27+0000 [-]   File "/usr/local/lib64/python3.8/site-packages/sqlalchemy/engine/base.py", line 3215, in _wrap_pool_connect
[service:api] 2022-09-09 17:20:27+0000 [-]     Connection._handle_dbapi_exception_noconnection(
[service:api] 2022-09-09 17:20:27+0000 [-]   File "/usr/local/lib64/python3.8/site-packages/sqlalchemy/engine/base.py", line 2069, in _handle_dbapi_exception_noconnection
[service:api] 2022-09-09 17:20:27+0000 [-]     util.raise_(
[service:api] 2022-09-09 17:20:27+0000 [-]   File "/usr/local/lib64/python3.8/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
[service:api] 2022-09-09 17:20:27+0000 [-]     raise exception
[service:api] 2022-09-09 17:20:27+0000 [-]   File "/usr/local/lib64/python3.8/site-packages/sqlalchemy/engine/base.py", line 3212, in _wrap_pool_connect
[service:api] 2022-09-09 17:20:27+0000 [-]     return fn()
[service:api] 2022-09-09 17:20:27+0000 [-]   File "/usr/local/lib64/python3.8/site-packages/sqlalchemy/pool/base.py", line 307, in connect
[service:api] 2022-09-09 17:20:27+0000 [-]     return _ConnectionFairy._checkout(self)
[service:api] 2022-09-09 17:20:27+0000 [-]   File "/usr/local/lib64/python3.8/site-packages/sqlalchemy/pool/base.py", line 767, in _checkout
[service:api] 2022-09-09 17:20:27+0000 [-]     fairy = _ConnectionRecord.checkout(pool)
[service:api] 2022-09-09 17:20:27+0000 [-]   File "/usr/local/lib64/python3.8/site-packages/sqlalchemy/pool/base.py", line 425, in checkout
[service:api] 2022-09-09 17:20:27+0000 [-]     rec = pool._do_get()
[service:api] 2022-09-09 17:20:27+0000 [-]   File "/usr/local/lib64/python3.8/site-packages/sqlalchemy/pool/impl.py", line 146, in _do_get
[service:api] 2022-09-09 17:20:27+0000 [-]     self._dec_overflow()
[service:api] 2022-09-09 17:20:27+0000 [-]   File "/usr/local/lib64/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
[service:api] 2022-09-09 17:20:27+0000 [-]     compat.raise_(
[service:api] 2022-09-09 17:20:27+0000 [-]   File "/usr/local/lib64/python3.8/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
[service:api] 2022-09-09 17:20:27+0000 [-]     raise exception
[service:api] 2022-09-09 17:20:27+0000 [-]   File "/usr/local/lib64/python3.8/site-packages/sqlalchemy/pool/impl.py", line 143, in _do_get
[service:api] 2022-09-09 17:20:27+0000 [-]     return self._create_connection()
[service:api] 2022-09-09 17:20:27+0000 [-]   File "/usr/local/lib64/python3.8/site-packages/sqlalchemy/pool/base.py", line 253, in _create_connection
[service:api] 2022-09-09 17:20:27+0000 [-]     return _ConnectionRecord(self)
[service:api] 2022-09-09 17:20:27+0000 [-]   File "/usr/local/lib64/python3.8/site-packages/sqlalchemy/pool/base.py", line 368, in __init__
[service:api] 2022-09-09 17:20:27+0000 [-]     self.__connect()
[service:api] 2022-09-09 17:20:27+0000 [-]   File "/usr/local/lib64/python3.8/site-packages/sqlalchemy/pool/base.py", line 611, in __connect
[service:api] 2022-09-09 17:20:27+0000 [-]     pool.logger.debug("Error on connect(): %s", e)
[service:api] 2022-09-09 17:20:27+0000 [-]   File "/usr/local/lib64/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
[service:api] 2022-09-09 17:20:27+0000 [-]     compat.raise_(
[service:api] 2022-09-09 17:20:27+0000 [-]   File "/usr/local/lib64/python3.8/site-packages/sqlalchemy/util/compat.py", line 207, in raise_

Catalog

[service:catalog] 2022-09-09 16:49:34+0000 [-] [Thread-22] [anchore_engine.subsys.servicestatus/handle_service_heartbeat()] [ERROR] (psycopg2.OperationalError) timeout expired
[service:catalog] 	
[service:catalog] 	(Background on this error at: https://sqlalche.me/e/14/e3q8)

Policy:

[service:policy-engine] 2022-09-09 16:32:51+0000 [-] [Thread-11] [anchore_engine.utils/timer()] [INFO] Execution of image vulnerabilities refresh for admin/2e7e4cd3a0104daee76278b1f171e6d030a22d9fc3c4b4a0af6fbd4156eb5d81 took: 226.83438849449158 seconds
[service:policy-engine] 2022-09-09 16:32:51+0000 [-] Traceback (most recent call last):
[service:policy-engine] 2022-09-09 16:32:51+0000 [-]   File "/usr/local/lib/python3.8/site-packages/anchore_engine/services/policy_engine/__init__.py", line 451, in handle_image_vulnerabilities_refresh
[service:policy-engine] 2022-09-09 16:32:51+0000 [-]     ImageVulnerabilitiesRefreshTask(message).execute()
[service:policy-engine] 2022-09-09 16:32:51+0000 [-]   File "/usr/local/lib/python3.8/site-packages/anchore_engine/services/policy_engine/engine/tasks.py", line 627, in execute
[service:policy-engine] 2022-09-09 16:32:51+0000 [-]     get_vulnerabilities_provider().get_image_vulnerabilities_json(
[service:policy-engine] 2022-09-09 16:32:51+0000 [-]   File "/usr/local/lib/python3.8/site-packages/anchore_engine/services/policy_engine/engine/vulns/providers.py", line 1129, in get_image_vulnerabilities_json
[service:policy-engine] 2022-09-09 16:32:51+0000 [-]     report = self._try_load_report_from_store(image, db_session)
[service:policy-engine] 2022-09-09 16:32:51+0000 [-]   File "/usr/local/lib/python3.8/site-packages/anchore_engine/services/policy_engine/engine/vulns/providers.py", line 1261, in _try_load_report_from_store
[service:policy-engine] 2022-09-09 16:32:51+0000 [-]     new_report = self.__scanner__().scan_image_for_vulnerabilities(
[service:policy-engine] 2022-09-09 16:32:51+0000 [-]   File "/usr/local/lib/python3.8/site-packages/anchore_engine/services/policy_engine/engine/vulns/scanners.py", line 225, in scan_image_for_vulnerabilities
[service:policy-engine] 2022-09-09 16:32:51+0000 [-]     GrypeDBSyncManager.run_grypedb_sync(db_session)
[service:policy-engine] 2022-09-09 16:32:51+0000 [-]   File "/usr/local/lib/python3.8/site-packages/anchore_engine/services/policy_engine/engine/feeds/grypedb_sync.py", line 176, in run_grypedb_sync
[service:policy-engine] 2022-09-09 16:32:51+0000 [-]     with GrypeDBSyncLock(LOCK_AQUISITION_TIMEOUT):
[service:policy-engine] 2022-09-09 16:32:51+0000 [-]   File "/usr/local/lib/python3.8/site-packages/anchore_engine/services/policy_engine/engine/feeds/grypedb_sync.py", line 51, in __enter__
[service:policy-engine] 2022-09-09 16:32:51+0000 [-]     raise GrypeDBSyncLockAquisitionTimeout(self.timeout)
[service:policy-engine] 2022-09-09 16:32:51+0000 [-] anchore_engine.services.policy_engine.engine.feeds.grypedb_sync.GrypeDBSyncLockAquisitionTimeout: Aquisition timeout of 60 seconds encountered before lock was released. Potential deadlock in system.

Simple Queue

[service:simplequeue] 2022-09-09 13:29:20+0000 [_GenericHTTPChannelProtocol,1527,100.102.82.187] [PoolThread-twisted.internet.reactor-9] [anchore_engine.db.entities.exceptions/_get_pgcode_from_ex()] [WARN] cannot extract PG code from driver exception - exception details: Lock no longer held by this id
[service:simplequeue] 2022-09-09 13:29:20+0000 [_GenericHTTPChannelProtocol,1528,100.102.82.187] [PoolThread-twisted.internet.reactor-5] [anchore_engine.db.entities.exceptions/_get_pgcode_from_ex()] [WARN] cannot extract PG code from driver exception - exception details: Lock no longer held by this id
[service:simplequeue] 2022-09-09 13:29:20+0000 [-] Traceback (most recent call last):
[service:simplequeue] 2022-09-09 13:29:20+0000 [-] Traceback (most recent call last):
[service:simplequeue] 2022-09-09 13:29:20+0000 [-]   File "/usr/local/lib/python3.8/site-packages/anchore_engine/db/db_locks.py", line 217, in refresh_lease
[service:simplequeue] 2022-09-09 13:29:20+0000 [-]   File "/usr/local/lib/python3.8/site-packages/anchore_engine/db/db_locks.py", line 217, in refresh_lease
[service:simplequeue] 2022-09-09 13:29:20+0000 [-]     raise Exception("Lock no longer held by this id")
[service:simplequeue] 2022-09-09 13:29:20+0000 [-]     raise Exception("Lock no longer held by this id")
[service:simplequeue] 2022-09-09 13:29:20+0000 [-] Exception: Lock no longer held by this id
[service:simplequeue] 2022-09-09 13:29:20+0000 [-] Exception: Lock no longer held by this id
[service:simplequeue] 2022-09-09 13:29:20+0000 [_GenericHTTPChannelProtocol,1529,100.102.82.187] [PoolThread-twisted.internet.reactor-11] [anchore_engine.db.entities.exceptions/_get_pgcode_from_ex()] [WARN] cannot extract PG code from driver exception - exception details: Lock no longer held by this id
[service:simplequeue] 2022-09-09 13:29:20+0000 [-] Traceback (most recent call last):
[service:simplequeue] 2022-09-09 13:29:20+0000 [-]   File "/usr/local/lib/python3.8/site-packages/anchore_engine/db/db_locks.py", line 217, in refresh_lease
[service:simplequeue] 2022-09-09 13:29:20+0000 [-]     raise Exception("Lock no longer held by this id")
[service:simplequeue] 2022-09-09 13:29:20+0000 [-] Exception: Lock no longer held by this id
[service:simplequeue] 2022-09-09 13:29:20+0000 [_GenericHTTPChannelProtocol,1527,100.102.82.187] [PoolThread-twisted.internet.reactor-9] [anchore_engine.db.db_locks/refresh_lease()] [ERROR] Failed updating lease duration for image_gc due to exception
[service:simplequeue] 2022-09-09 13:29:20+0000 [_GenericHTTPChannelProtocol,1529,100.102.82.187] [PoolThread-twisted.internet.reactor-11] [anchore_engine.db.db_locks/refresh_lease()] [ERROR] Failed updating lease duration for image_gc due to exception
[service:simplequeue] 2022-09-09 13:29:20+0000 [_GenericHTTPChannelProtocol,1528,100.102.82.187] [PoolThread-twisted.internet.reactor-5] [anchore_engine.db.db_locks/refresh_lease()] [ERROR] Failed updating lease duration for image_gc due to exception
[service:simplequeue] 2022-09-09 13:29:20+0000 [_GenericHTTPChannelProtocol,1527,100.102.82.187] [PoolThread-twisted.internet.reactor-9] [anchore_engine.db.entities.exceptions/_get_pgcode_from_ex()] [WARN] cannot extract PG code from driver exception - exception details: Lock no longer held by this id
[service:simplequeue] 2022-09-09 13:29:20+0000 [-] Traceback (most recent call last):
[service:simplequeue] 2022-09-09 13:29:20+0000 [-]   File "/usr/local/lib/python3.8/site-packages/anchore_engine/db/db_locks.py", line 217, in refresh_lease
[service:simplequeue] 2022-09-09 13:29:20+0000 [-]     raise Exception("Lock no longer held by this id")
[service:simplequeue] 2022-09-09 13:29:20+0000 [-] Exception: Lock no longer held by this id
[service:simplequeue] 2022-09-09 13:29:20+0000 [_GenericHTTPChannelProtocol,1529,100.102.82.187] [PoolThread-twisted.internet.reactor-11] [anchore_engine.db.entities.exceptions/_get_pgcode_from_ex()] [WARN] cannot extract PG code from driver exception - exception details: Lock no longer held by this id
[service:simplequeue] 2022-09-09 13:29:20+0000 [-] Traceback (most recent call last):
[service:simplequeue] 2022-09-09 13:29:20+0000 [-]   File "/usr/local/lib/python3.8/site-packages/anchore_engine/db/db_locks.py", line 217, in refresh_lease
[service:simplequeue] 2022-09-09 13:29:20+0000 [-]     raise Exception("Lock no longer held by this id")
[service:simplequeue] 2022-09-09 13:29:20+0000 [_GenericHTTPChannelProtocol,1528,100.102.82.187] [PoolThread-twisted.internet.reactor-5] [anchore_engine.db.entities.exceptions/_get_pgcode_from_ex()] [WARN] cannot extract PG code from driver exception - exception details: Lock no longer held by this id
[service:simplequeue] 2022-09-09 13:29:20+0000 [-] Exception: Lock no longer held by this id
[service:simplequeue] 2022-09-09 13:29:20+0000 [-] Traceback (most recent call last):
[service:simplequeue] 2022-09-09 13:29:20+0000 [_GenericHTTPChannelProtocol,1527,100.102.82.187] [PoolThread-twisted.internet.reactor-9] [anchore_engine.db.db_locks/refresh_lease()] [ERROR] Failed updating lease duration for image_gc due to exception
[service:simplequeue] 2022-09-09 13:29:20+0000 [-]   File "/usr/local/lib/python3.8/site-packages/anchore_engine/db/db_locks.py", line 217, in refresh_lease
[service:simplequeue] 2022-09-09 13:29:20+0000 [-]     raise Exception("Lock no longer held by this id")
[service:simplequeue] 2022-09-09 13:29:20+0000 [-] Exception: Lock no longer held by this id
[service:simplequeue] 2022-09-09 13:29:20+0000 [_GenericHTTPChannelProtocol,1529,100.102.82.187] [PoolThread-twisted.internet.reactor-11] [anchore_engine.db.db_locks/refresh_lease()] [ERROR] Failed updating lease duration for image_gc due to exception
[service:simplequeue] 2022-09-09 13:29:20+0000 [_GenericHTTPChannelProtocol,1527,100.102.82.187] [PoolThread-twisted.internet.reactor-9] [anchore_engine.db.entities.exceptions/_get_pgcode_from_ex()] [WARN] cannot extract PG code from driver exception - exception details: Lock no longer held by this id
[service:simplequeue] 2022-09-09 13:29:20+0000 [-] Traceback (most recent call last):
[service:simplequeue] 2022-09-09 13:29:20+0000 [-]   File "/usr/local/lib/python3.8/site-packages/anchore_engine/db/db_locks.py", line 217, in refresh_lease
[service:simplequeue] 2022-09-09 13:29:20+0000 [-]     raise Exception("Lock no longer held by this id")
[service:simplequeue] 2022-09-09 13:29:20+0000 [-] Exception: Lock no longer held by this id
[service:simplequeue] 2022-09-09 13:29:20+0000 [_GenericHTTPChannelProtocol,1529,100.102.82.187] [PoolThread-twisted.internet.reactor-11] [anchore_engine.db.db_locks/refresh_lease()] [ERROR] Failed updating lease duration image_gc after all retries
[service:simplequeue] 2022-09-09 13:29:20+0000 [-] "100.102.82.187" - - [09/Sep/2022:13:29:20 +0000] "PUT /v1/leases/image_gc/ttl?client_id=anchore-anchore-engine-catalog-785d79dcb6-dj7mf%3A27%3A139886254343936%3A&ttl=60&epoch=870805 HTTP/1.1" 200 - "-" "python-requests/2.26.0"
[service:simplequeue] 2022-09-09 13:29:20+0000 [_GenericHTTPChannelProtocol,1528,100.102.82.187] [PoolThread-twisted.internet.reactor-5] [anchore_engine.db.db_locks/refresh_lease()] [ERROR] Failed updating lease duration for image_gc due to exception
[service:simplequeue] 2022-09-09 13:29:20+0000 [_GenericHTTPChannelProtocol,1527,100.102.82.187] [PoolThread-twisted.internet.reactor-9] [anchore_engine.db.db_locks/refresh_lease()] [ERROR] Failed updating lease duration for image_gc due to exception
[service:simplequeue] 2022-09-09 13:29:20+0000 [_GenericHTTPChannelProtocol,1527,100.102.82.187] [PoolThread-twisted.internet.reactor-9] [anchore_engine.db.db_locks/refresh_lease()] [ERROR] Failed updating lease duration image_gc after all retries
[service:simplequeue] 2022-09-09 13:29:20+0000 [-] "100.102.82.187" - - [09/Sep/2022:13:29:20 +0000] "PUT /v1/leases/image_gc/ttl?client_id=anchore-anchore-engine-catalog-785d79dcb6-dj7mf%3A27%3A139887395194624%3A&ttl=60&epoch=871322 HTTP/1.1" 200 - "-" "python-requests/2.26.0"
[service:simplequeue] 2022-09-09 13:29:20+0000 [_GenericHTTPChannelProtocol,1528,100.102.82.187] [PoolThread-twisted.internet.reactor-5] [anchore_engine.db.db_locks/refresh_lease()] [ERROR] Failed updating lease duration image_gc after all retries

What docker images are you using:
v1.0.0

Anything else we need to know:
Helm Deployment values;

  values:
    anchoreGlobal:
      image: docker.io/anchore/anchore-engine:v1.0.0
      certStoreSecretName: anchore-cert
      defaultAdminPassword: *****
      defaultAdminEmail: *****
    ingress:
      enabled: true
      apiHosts:
        - *****
      annotations:
        kubernetes.io/ingress.class: nginx
        cert-manager.io/cluster-issuer: letsencrypt-prod
      tls: 
      # Secrets must be manually created in the namespace.
       - secretName: anchore-api-cert
         hosts:
           - *****
    anchorePolicyEngine:  
      vulnerabilityProvider: grype 
      resources:
       requests:
         cpu: 400m
         memory: 1Gi
    anchoreAnalyzer:
      resources:
       requests:
         cpu: 400m
         memory: 1Gi
    anchoreApi:
      resources:
       requests:
         cpu: 100m
         memory: 1Gi
    anchoreCatalog:
      resources:
       requests:
         cpu: 100m
         memory: 500Mi
    anchoreSimpleQueue:
      resources:
         #memory: 1Gi
       requests:
         cpu: 100m
         memory: 256Mi
    postgresql:
      enabled: false
      externalEndpoint: *****
      postgresUser: *****
      postgresPassword: *****
      postgresDatabase: anchore 
      persistence:
        existingclaim: anchore-postgresql
        resourcePolicy: nil
        size: 50Gi

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions