Skip to content
This repository was archived by the owner on Apr 1, 2023. It is now read-only.
This repository was archived by the owner on Apr 1, 2023. It is now read-only.

HTTP proxy assertion fail on shutdown with ongoing txns #310

@iafuture

Description

@iafuture

It seems the heartbeat for a txn is not cancelled when the txn is dropped in the HTTPProxy, causing the hb timer to attempt to use the RPCDispatcher to send a message. The dispatcher has shutdown at this point and asserts that it should not be used, e.g.

[0004:14:18:51.953.116]-http_proxy-[0]-(k2::skv_client) [DEBUG] [/build/src/k2/module/k23si/client/k23si_client.cpp:66 @operator()] send hb for mtr={timestamp={endCount=397131397989158, tsoId=100000, startDelta=100000}, priority=medium}
[0004:14:18:51.953.201]-http_proxy-[0]-(k2::cpo_client) [DEBUG] [/build/src/k2/cpo/client/Client.h:88 @partitionRequest] making partition request with deadline=0000:00:00:00.499.993
[0004:14:18:52.009.280]-http_proxy-[0]-(k2::transport) [ERROR] [/build/src/k2/transport/RPCDispatcher.h:189 @operator()] RPC send failed with uncaught exception: dispatcher has shut down
[0004:14:18:52.009.353]-http_proxy-[0]-(k2::cpo_client) [DEBUG] [/build/src/k2/cpo/client/Client.h:360 @operator()] partition call completed with status={code=500, message=unknown exception while sending request}
[0004:14:18:52.009.405]-http_proxy-[0]-(k2::cpo_client) [DEBUG] [/build/src/k2/cpo/client/Client.h:105 @operator()] call failed with retryable status: {code=503, message=unable to make call at this time}
[0004:14:18:52.009.423]-http_proxy-[0]-(k2::cpo_client) [DEBUG] [/build/src/k2/cpo/client/Client.h:110 @operator()] refreshing collection from CPO after status={code=503, message=unable to make call at this time}
[0004:14:18:52.010.636]-http_proxy-[0]-(k2::cpo_client) [DEBUG] [/build/src/k2/cpo/client/Client.h:247 @_getAssignedPartitionWithRetry] time remaining=0000:00:00:00.442.563, for coll=HTTPClient, key={schemaName=test_schema, partitionKey=^00^00^01, rangeKey=^00^00^01}, reverse=false, excludedKey=false, retries=1
[0004:14:18:52.013.215]-http_proxy-[0]-(k2::cpo_client) [DEBUG] [/build/src/k2/cpo/client/Client.h:273 @_getAssignedPartitionWithRetry] no existing waiter for name=HTTPClient. Creating new one
http_proxy: /usr/local/include/seastar/core/sharded.hh:755: Service& seastar::sharded::local() [with Service = k2::RPCDispatcher]: Assertion `local_is_initialized()' failed.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions