[fix][broker] rackaware policy is ineffective when delete zk rack info after bkclient initialize by TakaHiR07 · Pull Request #20944 · apache/pulsar

TakaHiR07 · 2023-08-07T08:09:52Z

Motivation

#14248 fix rackaware placement policy does not take effect after delete rack configuration. But after #16825, this fix is ineffective when delete zk rack info after bkclient initialize.

The reason is "register available bookie" is after "BookieRackAffinityMapping#setConf" in bookieClient constructor. So updateRacksWithHost(racksWithHost) would throw BookieIdNotResolvedException, make "racksWithHost" become null. "racksWithHost" is updated until "watchAvailableBookies()" listener is trigger.

I add some log of unittest testRackUpdate() to show the order of bookieClient constructor:

2023-08-07T15:47:23,540+0800 [main] INFO  org.apache.pulsar.bookie.rackawareness.BookieRackAffinityMapping - ZkBookieRackAffinityMapping setConf(), racksWithHost:{default={10.0.0.1:56962=BookieInfoImpl(rack=rack-0, hostname=bookie-1), 10.0.0.2:56964=BookieInfoImpl(rack=rack-0, hostname=bookie-2), 10.0.0.3:56966=BookieInfoImpl(rack=rack-0, hostname=bookie-3), 10.0.0.4:56969=BookieInfoImpl(rack=rack-1, hostname=bookie-4), 10.0.0.5:56971=BookieInfoImpl(rack=rack-1, hostname=bookie-5), 10.0.0.6:56973=BookieInfoImpl(rack=rack-1, hostname=bookie-6)}}
2023-08-07T15:47:23,540+0800 [main] INFO  org.apache.bookkeeper.client.DefaultBookieAddressResolver - Cannot resolve 10.0.0.1:56962, bookie is unknown org.apache.bookkeeper.client.BKException$BKBookieHandleNotAvailableException: Bookie handle is not available
2023-08-07T15:47:23,540+0800 [main] INFO  org.apache.pulsar.bookie.rackawareness.BookieRackAffinityMapping - Network address for 10.0.0.1:56962 is unresolvable yet. error is org.apache.bookkeeper.proto.BookieAddressResolver$BookieIdNotResolvedException: Cannot resolve bookieId 10.0.0.1:56962, bookie does not exist or it is not running
2023-08-07T15:47:23,540+0800 [main] INFO  org.apache.bookkeeper.client.DefaultBookieAddressResolver - Cannot resolve 10.0.0.2:56964, bookie is unknown org.apache.bookkeeper.client.BKException$BKBookieHandleNotAvailableException: Bookie handle is not available
2023-08-07T15:47:23,540+0800 [main] INFO  org.apache.pulsar.bookie.rackawareness.BookieRackAffinityMapping - Network address for 10.0.0.2:56964 is unresolvable yet. error is org.apache.bookkeeper.proto.BookieAddressResolver$BookieIdNotResolvedException: Cannot resolve bookieId 10.0.0.2:56964, bookie does not exist or it is not running
2023-08-07T15:47:23,540+0800 [main] INFO  org.apache.bookkeeper.client.DefaultBookieAddressResolver - Cannot resolve 10.0.0.3:56966, bookie is unknown org.apache.bookkeeper.client.BKException$BKBookieHandleNotAvailableException: Bookie handle is not available
2023-08-07T15:47:23,540+0800 [main] INFO  org.apache.pulsar.bookie.rackawareness.BookieRackAffinityMapping - Network address for 10.0.0.3:56966 is unresolvable yet. error is org.apache.bookkeeper.proto.BookieAddressResolver$BookieIdNotResolvedException: Cannot resolve bookieId 10.0.0.3:56966, bookie does not exist or it is not running
2023-08-07T15:47:23,540+0800 [main] INFO  org.apache.bookkeeper.client.DefaultBookieAddressResolver - Cannot resolve 10.0.0.4:56969, bookie is unknown org.apache.bookkeeper.client.BKException$BKBookieHandleNotAvailableException: Bookie handle is not available
2023-08-07T15:47:23,540+0800 [main] INFO  org.apache.pulsar.bookie.rackawareness.BookieRackAffinityMapping - Network address for 10.0.0.4:56969 is unresolvable yet. error is org.apache.bookkeeper.proto.BookieAddressResolver$BookieIdNotResolvedException: Cannot resolve bookieId 10.0.0.4:56969, bookie does not exist or it is not running
2023-08-07T15:47:23,541+0800 [main] INFO  org.apache.bookkeeper.client.DefaultBookieAddressResolver - Cannot resolve 10.0.0.5:56971, bookie is unknown org.apache.bookkeeper.client.BKException$BKBookieHandleNotAvailableException: Bookie handle is not available
2023-08-07T15:47:23,541+0800 [main] INFO  org.apache.pulsar.bookie.rackawareness.BookieRackAffinityMapping - Network address for 10.0.0.5:56971 is unresolvable yet. error is org.apache.bookkeeper.proto.BookieAddressResolver$BookieIdNotResolvedException: Cannot resolve bookieId 10.0.0.5:56971, bookie does not exist or it is not running
2023-08-07T15:47:23,541+0800 [main] INFO  org.apache.bookkeeper.client.DefaultBookieAddressResolver - Cannot resolve 10.0.0.6:56973, bookie is unknown org.apache.bookkeeper.client.BKException$BKBookieHandleNotAvailableException: Bookie handle is not available
2023-08-07T15:47:23,541+0800 [main] INFO  org.apache.pulsar.bookie.rackawareness.BookieRackAffinityMapping - Network address for 10.0.0.6:56973 is unresolvable yet. error is org.apache.bookkeeper.proto.BookieAddressResolver$BookieIdNotResolvedException: Cannot resolve bookieId 10.0.0.6:56973, bookie does not exist or it is not running
2023-08-07T15:47:23,541+0800 [main] INFO  org.apache.pulsar.bookie.rackawareness.BookieRackAffinityMapping - ZkBookieRackAffinityMapping setConf() after updateRacksWithHost(), racksWithHost:{}
2023-08-07T15:47:23,541+0800 [main] WARN  org.apache.bookkeeper.client.TopologyAwareEnsemblePlacementPolicy - Failed to resolve network location for 127.0.0.1, using default rack for it : /default-rack.
2023-08-07T15:47:23,541+0800 [main] INFO  org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Initialize rackaware ensemble placement policy @ <Bookie:127.0.0.1:0> @ /default-rack : org.apache.pulsar.bookie.rackawareness.BookieRackAffinityMapping.
2023-08-07T15:47:23,541+0800 [main] INFO  org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Not weighted
2023-08-07T15:47:23,541+0800 [main] INFO  org.apache.bookkeeper.client.BookKeeper - Weighted ledger placement is not enabled
2023-08-07T15:47:23,545+0800 [metadata-store-172-1] INFO  org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient - Update BookieInfoCache (writable bookie) 10.0.0.5:56971 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=10.0.0.5:56971, port=56971, host=10.0.0.5, protocol=bookie-rpc, auth=[], extensions=[]}]}
2023-08-07T15:47:23,545+0800 [metadata-store-172-1] INFO  org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient - Update BookieInfoCache (writable bookie) 10.0.0.5:56971 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=10.0.0.5:56971, port=56971, host=10.0.0.5, protocol=bookie-rpc, auth=[], extensions=[]}]}
2023-08-07T15:47:23,545+0800 [metadata-store-172-1] INFO  org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient - Update BookieInfoCache (writable bookie) 10.0.0.2:56964 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=10.0.0.2:56964, port=56964, host=10.0.0.2, protocol=bookie-rpc, auth=[], extensions=[]}]}
2023-08-07T15:47:23,545+0800 [metadata-store-172-1] INFO  org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient - Update BookieInfoCache (writable bookie) 10.0.0.2:56964 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=10.0.0.2:56964, port=56964, host=10.0.0.2, protocol=bookie-rpc, auth=[], extensions=[]}]}
2023-08-07T15:47:23,545+0800 [metadata-store-172-1] INFO  org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient - Update BookieInfoCache (writable bookie) 10.0.0.6:56973 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=10.0.0.6:56973, port=56973, host=10.0.0.6, protocol=bookie-rpc, auth=[], extensions=[]}]}
2023-08-07T15:47:23,545+0800 [metadata-store-172-1] INFO  org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient - Update BookieInfoCache (writable bookie) 10.0.0.6:56973 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=10.0.0.6:56973, port=56973, host=10.0.0.6, protocol=bookie-rpc, auth=[], extensions=[]}]}
2023-08-07T15:47:23,545+0800 [metadata-store-172-1] INFO  org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient - Update BookieInfoCache (writable bookie) 10.0.0.4:56969 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=10.0.0.4:56969, port=56969, host=10.0.0.4, protocol=bookie-rpc, auth=[], extensions=[]}]}
2023-08-07T15:47:23,545+0800 [metadata-store-172-1] INFO  org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient - Update BookieInfoCache (writable bookie) 10.0.0.4:56969 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=10.0.0.4:56969, port=56969, host=10.0.0.4, protocol=bookie-rpc, auth=[], extensions=[]}]}
2023-08-07T15:47:23,545+0800 [metadata-store-172-1] INFO  org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient - Update BookieInfoCache (writable bookie) 10.0.0.3:56966 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=10.0.0.3:56966, port=56966, host=10.0.0.3, protocol=bookie-rpc, auth=[], extensions=[]}]}
2023-08-07T15:47:23,545+0800 [metadata-store-172-1] INFO  org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient - Update BookieInfoCache (writable bookie) 10.0.0.3:56966 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=10.0.0.3:56966, port=56966, host=10.0.0.3, protocol=bookie-rpc, auth=[], extensions=[]}]}
2023-08-07T15:47:23,545+0800 [metadata-store-172-1] INFO  org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient - Update BookieInfoCache (writable bookie) 10.0.0.1:56962 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=10.0.0.1:56962, port=56962, host=10.0.0.1, protocol=bookie-rpc, auth=[], extensions=[]}]}
2023-08-07T15:47:23,545+0800 [metadata-store-172-1] INFO  org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient - Update BookieInfoCache (writable bookie) 10.0.0.1:56962 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=10.0.0.1:56962, port=56962, host=10.0.0.1, protocol=bookie-rpc, auth=[], extensions=[]}]}
2023-08-07T15:47:23,545+0800 [pulsar-registration-client-209-1] INFO  org.apache.pulsar.bookie.rackawareness.BookieRackAffinityMapping - watchAvailableBookies() trigger racksWithHost:{default={10.0.0.1:56962=BookieInfoImpl(rack=rack-0, hostname=bookie-1), 10.0.0.2:56964=BookieInfoImpl(rack=rack-0, hostname=bookie-2), 10.0.0.3:56966=BookieInfoImpl(rack=rack-0, hostname=bookie-3), 10.0.0.4:56969=BookieInfoImpl(rack=rack-1, hostname=bookie-4), 10.0.0.5:56971=BookieInfoImpl(rack=rack-1, hostname=bookie-5), 10.0.0.6:56973=BookieInfoImpl(rack=rack-1, hostname=bookie-6)}}
2023-08-07T15:47:23,547+0800 [pulsar-registration-client-209-1] INFO  org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node: /rack-1/10.0.0.5:56971
2023-08-07T15:47:23,547+0800 [pulsar-registration-client-209-1] INFO  org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node: /rack-0/10.0.0.2:56964
2023-08-07T15:47:23,547+0800 [pulsar-registration-client-209-1] INFO  org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node: /rack-1/10.0.0.6:56973
2023-08-07T15:47:23,547+0800 [pulsar-registration-client-209-1] INFO  org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node: /rack-1/10.0.0.4:56969
2023-08-07T15:47:23,547+0800 [pulsar-registration-client-209-1] INFO  org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node: /rack-0/10.0.0.3:56966
2023-08-07T15:47:23,547+0800 [pulsar-registration-client-209-1] INFO  org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node: /rack-0/10.0.0.1:56962

Modifications

modify the execute order in BookieRackAffinityMapping#setConf
improve the testRackUpdate() to test this case.

Verifying this change

Make sure that the change passes the CI checks.

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

Documentation

doc
doc-required
doc-not-needed
doc-complete

Matching PR in forked repository

PR in forked repository: TakaHiR07#12

TakaHiR07 · 2023-08-14T09:24:32Z

@michaeljmarshall @horizonzy @hangc0276 @aloyszhang PTAL.

github-actions · 2023-09-14T01:46:58Z

The pr had no activity for 30 days, mark with Stale label.

horizonzy

LGTM.

horizonzy · 2023-09-20T04:08:32Z

/pulsarbot run-failure-checks

codecov-commenter · 2023-09-20T04:45:11Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 36.80%. Comparing base (57fbee4) to head (8930c21).
⚠️ Report is 1778 commits behind head on master.

Additional details and impacted files

@@              Coverage Diff              @@
##             master   #20944       +/-   ##
=============================================
- Coverage     72.97%   36.80%   -36.18%     
+ Complexity    32157      374    -31783     
=============================================
  Files          1868     1698      -170     
  Lines        139164   130430     -8734     
  Branches      15314    14250     -1064     
=============================================
- Hits         101555    47999    -53556     
- Misses        29562    76104    +46542     
+ Partials       8047     6327     -1720

Flag	Coverage Δ
inttests	`24.16% <100.00%> (?)`
systests	`24.69% <100.00%> (-0.36%)`	⬇️
unittests	`31.95% <100.00%> (-40.43%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...ookie/rackawareness/BookieRackAffinityMapping.java	`70.00% <100.00%> (-10.72%)`	⬇️

... and 1453 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…o after bkclient initialize (#20944) (cherry picked from commit d9ebaf5)

…o after bkclient initialize (apache#20944)

…o after bkclient initialize (#20944)

…o after bkclient initialize (apache#20944) (cherry picked from commit d9ebaf5)

…o after bkclient initialize (apache#20944)

…o after bkclient initialize (apache#20944) (cherry picked from commit d9ebaf5)

github-actions bot added the doc-not-needed Your PR changes do not impact docs label Aug 7, 2023

github-actions bot added the Stale label Sep 14, 2023

fix rackaware policy is ineffective after delete zk rack info

8930c21

TakaHiR07 force-pushed the fix_rackaware_policy_ineffective_after_delete_zk_rackInfo branch from e60964b to 8930c21 Compare September 19, 2023 08:37

hangc0276 requested review from codelipenghui, eolivelli, hangc0276, mattisonchao, michaeljmarshall, poorbarcode and zymap September 19, 2023 08:42

hangc0276 assigned TakaHiR07 Sep 19, 2023

hangc0276 added this to the 3.2.0 milestone Sep 19, 2023

hangc0276 approved these changes Sep 20, 2023

View reviewed changes

horizonzy approved these changes Sep 20, 2023

View reviewed changes

poorbarcode approved these changes Sep 20, 2023

View reviewed changes

codelipenghui approved these changes Oct 7, 2023

View reviewed changes

codelipenghui merged commit d9ebaf5 into apache:master Oct 7, 2023

poorbarcode pushed a commit that referenced this pull request Oct 7, 2023

[fix][broker] rackaware policy is ineffective when delete zk rack inf…

3129621

…o after bkclient initialize (#20944) (cherry picked from commit d9ebaf5)

poorbarcode added the cherry-picked/branch-3.0 label Oct 7, 2023

liangyuanpeng pushed a commit to liangyuanpeng/pulsar that referenced this pull request Oct 11, 2023

[fix][broker] rackaware policy is ineffective when delete zk rack inf…

bd869e7

…o after bkclient initialize (apache#20944)

vinayakmalik95 pushed a commit to tmdc-io/pulsar that referenced this pull request Oct 12, 2023

[fix][broker] rackaware policy is ineffective when delete zk rack inf…

eb744f4

…o after bkclient initialize (apache#20944)

Technoboy- pushed a commit that referenced this pull request Oct 19, 2023

[fix][broker] rackaware policy is ineffective when delete zk rack inf…

dfc2084

…o after bkclient initialize (#20944)

Technoboy- added cherry-picked/branch-3.1 release/3.1.2 and removed release/3.1.1 labels Oct 19, 2023

shibd added the cherry-picked/branch-2.11 label Oct 22, 2023

shibd pushed a commit to shibd/pulsar that referenced this pull request Oct 22, 2023

[fix][broker] rackaware policy is ineffective when delete zk rack inf…

add98cd

…o after bkclient initialize (apache#20944) (cherry picked from commit d9ebaf5)

shibd pushed a commit to shibd/pulsar that referenced this pull request Oct 24, 2023

[fix][broker] rackaware policy is ineffective when delete zk rack inf…

79069f3

…o after bkclient initialize (apache#20944) (cherry picked from commit d9ebaf5)

shibd mentioned this pull request Oct 24, 2023

[Don't merge][cherry-pick][branch-2.11] Cherry-pick some PR to branch-2.11 #21427

Closed

14 tasks

nikhil-ctds pushed a commit to datastax/pulsar that referenced this pull request Dec 20, 2023

[fix][broker] rackaware policy is ineffective when delete zk rack inf…

c7035cf

…o after bkclient initialize (apache#20944)

srinath-ctds pushed a commit to datastax/pulsar that referenced this pull request Dec 20, 2023

[fix][broker] rackaware policy is ineffective when delete zk rack inf…

a1d3336

…o after bkclient initialize (apache#20944)

lhotari added release/2.10.7 and removed release/2.10.6 labels Mar 5, 2024

Meet0861 mentioned this pull request May 13, 2024

[Bug] Deadlock in broker service while initializing bkClient #22699

Closed

3 tasks

nodece pushed a commit to nodece/pulsar that referenced this pull request Aug 15, 2024

[fix][broker] rackaware policy is ineffective when delete zk rack inf…

31c6312

…o after bkclient initialize (apache#20944) (cherry picked from commit d9ebaf5)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[fix][broker] rackaware policy is ineffective when delete zk rack info after bkclient initialize#20944

[fix][broker] rackaware policy is ineffective when delete zk rack info after bkclient initialize#20944
codelipenghui merged 1 commit intoapache:masterfrom
TakaHiR07:fix_rackaware_policy_ineffective_after_delete_zk_rackInfo

TakaHiR07 commented Aug 7, 2023 •

edited

Loading

Uh oh!

TakaHiR07 commented Aug 14, 2023

Uh oh!

github-actions bot commented Sep 14, 2023

Uh oh!

horizonzy left a comment

Uh oh!

horizonzy commented Sep 20, 2023

Uh oh!

codecov-commenter commented Sep 20, 2023 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

Conversation

TakaHiR07 commented Aug 7, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

Matching PR in forked repository

Uh oh!

TakaHiR07 commented Aug 14, 2023

Uh oh!

github-actions bot commented Sep 14, 2023

Uh oh!

horizonzy left a comment

Choose a reason for hiding this comment

Uh oh!

horizonzy commented Sep 20, 2023

Uh oh!

codecov-commenter commented Sep 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

TakaHiR07 commented Aug 7, 2023 •

edited

Loading

codecov-commenter commented Sep 20, 2023 •

edited

Loading