Range retrieval #8103

thalman · 2025-09-19T14:02:00Z

Implement support for range retrieval of attribute values.
When a multi-valued attribute has a large number of values Active
Directory response only with first batch of values. A common case
is the memberOf attribute when a user belongs to many groups.
This is controlled by AD MaxValRange values (default 1500) policy.

Range retrieval is indicated in the response and provides information
which part of the batch is provided (e. g., memberOf;range=0-1499).

SSSD must then perform additional requests to fetch the remaining
values (e. g., memberOf;range=1500-*)..

The implementation is written to allow extension to other search types
(e. g., root DSE search), although that is not currently required.

Resolves: #8102

thalman · 2025-09-25T13:17:07Z

I realized that I need also update a man page. Coming soon as separate commit.

src/providers/ldap/sdap_async.c

src/providers/ldap/sdap.c

alexey-tikhonov · 2025-10-15T09:16:01Z

Looks like man: Update range retrieval information commit includes stray changes.

src/providers/ldap/sdap.c

thalman · 2025-10-21T20:11:51Z

Hi guys, I incorporated Alexey's comments. We also have a test, thanks @jakub-vavra-cz.

pbrezina

The failure in test_access_filter__ldap_attributes_approximately_greater_and_less_than_permits_user_login does not seem to be related.

pbrezina · 2025-11-04T14:04:28Z

@thalman The failure does not seem to be related, but the result is consistent after a retry and CI on master branch is green. Please, take a closer look at it.

alexey-tikhonov · 2025-11-04T17:01:10Z

src/tests/system/tests/test_ad.py

+    client.sssctl.cache_expire(everything=True)
+    client.host.conn.run(f"id -u big-user@{client.sssd.default_domain}")
+    grps = client.host.conn.run(
+        f"ldbsearch -H /var/lib/sss/db/cache_{client.sssd.default_domain}.ldb"


@jakub-vavra-cz, why do we need to read cache directly?
What's wrong with using id $user output?

id $user is too complex and runs more ldap searches. That can add missing parts of the object. The test was not working when we tried it that way, maybe even unstable (Am I right @jakub-vavra-cz?).

This way it provides stable and correct result.

Ok, I got it. This is because you look for originalMemberOf that is obtained as a multi-value attr.
getgrouplist() will use LDAP search by member/memberUid and will return a multiple objects instead, not what this test want.

I wonder if originalMemberOf is actually used at all.
getgrouplist() for AD is usually obtained via token-groups search.

So perhaps this test tests something that is not important.
Relies on an internal implementation details and thus prone to be broken.

alexey-tikhonov · 2025-11-04T17:03:24Z

src/tests/system/tests/test_ad.py

+            """
+    )
+    client.sssctl.cache_expire(everything=True)
+    client.host.conn.run(f"id -u big-user@{client.sssd.default_domain}")


@jakub-vavra-cz , why not client.tools.id()?

And why -u?

the -u guaranties that we perform just one ldapsearch asking for user object. Not resolving all the group where user is member of.

This is also why we can't use client.tools.id()

But this will only call getpwnam() under the hood.

So just do client.tools.getent(), it will have the same effect and will avoid custom 'conn.run()'

The issue that I have with "high level tools" at the moment is that they work inconsistently with old version of SSSD. So if we break range retrieval, such test may or may not fail. This is why I decided to check cache content. At the moment I do not have reproducer for this issue that is stable.

I'm exploring some ideas to have the large group test case as well but I doubt that we can do more in this one.

The issue that I have with "high level tools" at the moment is that they work inconsistently with old version of SSSD. So if we break range retrieval, such test may or may not fail. This is why I decided to check cache content.

You check for the cache content because you check 'originalMemberOf' that is not exposed otherwise.

Still you can use client.tools.getent() because it is essentially the same as id -u.

If it is not the same then I miss something and would like to understand what is it.

alexey-tikhonov · 2025-11-04T17:04:23Z

src/tests/system/tests/test_ad.py

+@pytest.mark.importance("high")
+def test_range_retrieval__group_membership(client: Client, ad: AD):
+    """
+    :title: Users with with more groups than MaxValRange can be retrieved


@jakub-vavra-cz, could you please also add a test for a case "a group with more than MaxValRange users"?

@alexey-tikhonov I had that exact test initially as it is how I tried to reproduce the issue
but it was not working properly. The test was mostly stable and the failures happening randomly were not deterministic and could be attributed to large data flying over network on from under-speced machines.

We do not know at the moment how to test it reliably (the test sometimes succeeded with old SSSD version without range retrieval) but I will think of it. I suggest not to wait and do that in separate PR once we figure out how to actually test that.

Ok, then for a start, I need to understand something.

Does AD really store group membership in user objects directly?

IIUC, that's not the case with OpenLDAP/DS (at least by default).

Right:

In Active Directory, memberOf is a "virtual" or "back-link" attribute that the server calculates and maintains automatically.

could be attributed to large data flying over network

What is the difference '152 values of memberOf attr' vs '152 values of member/memberUid attr'?

Did you try using 'getent group'?

@alexey-tikhonov See: bc577fe
This is how I was trying to reproduce with group containing 1502 members, but the failures were random and inconsistent.

@jakub-vavra-cz , can you try the same but with size = 152?

Imo, this is more important use-case to cover than 'originalMemberOf' as the latter, likely, is not really used.

alexey-tikhonov · 2025-11-04T17:18:04Z

It would be great to have also @sumit-bose review for the most important patch - "sdap: implement range retrieval for generic search"

pbrezina

Thank you, ack.

src/providers/ldap/sdap_async.c

Implement support for range retrieval of attribute values. When a multi-valued attribute has a large number of values Active Directory response only with first batch of values. A common case is the `memberOf` attribute when a user belongs to many groups. This is controlled by AD `MaxValRange` values (default 1500) policy. Range retrieval is indicated in the response and provides information which part of the batch is provided (e. g., memberOf;range=0-1499). SSSD must then perform additional requests to fetch the remaining values (e. g., `memberOf;range=1500-*`).. The implementation is written to allow extension to other search types (e. g., root DSE search), although that is not currently required. Resolves: SSSD#8102

When performing range retrieval Active Directory guaranties that the follwing response does not contain overlaps (in the same connection). It is expensive to compare all the attribute values against each other. We can relay on AD and append the subsequent response without checking.

thalman · 2025-11-10T13:46:35Z

@pbrezina, @alexey-tikhonov - I incorporated comments and also fixed the crash that I introduced and was discovered by one test. Feel free to re-check

alexey-tikhonov · 2025-11-15T14:50:59Z

All the comments I have are around the test, but I'd like to have those addressed.

thalman force-pushed the range_retrieval branch from 7354639 to 108e728 Compare September 22, 2025 11:21

sdap: Create list of attributes for range retrieval

bbbf145

thalman force-pushed the range_retrieval branch from 108e728 to e8110d9 Compare September 22, 2025 12:00

thalman changed the title ~~Draft: Range retrieval~~ Range retrieval Sep 22, 2025

thalman marked this pull request as ready for review September 22, 2025 12:00

thalman force-pushed the range_retrieval branch from e8110d9 to 3073754 Compare September 24, 2025 09:24

alexey-tikhonov added Waiting for review backport-to-sssd-2-9 labels Sep 25, 2025

alexey-tikhonov self-requested a review September 25, 2025 12:35

alexey-tikhonov self-assigned this Sep 25, 2025

alexey-tikhonov requested a review from pbrezina September 25, 2025 12:35

alexey-tikhonov assigned pbrezina Sep 25, 2025

siteshwar reviewed Oct 9, 2025

View reviewed changes

src/providers/ldap/sdap_async.c Outdated Show resolved Hide resolved

alexey-tikhonov reviewed Oct 15, 2025

View reviewed changes

src/providers/ldap/sdap.c Outdated Show resolved Hide resolved

alexey-tikhonov reviewed Oct 15, 2025

View reviewed changes

src/providers/ldap/sdap.c Outdated Show resolved Hide resolved

alexey-tikhonov reviewed Oct 15, 2025

View reviewed changes

src/providers/ldap/sdap.c Show resolved Hide resolved

thalman force-pushed the range_retrieval branch 2 times, most recently from 8273797 to 3b10082 Compare October 21, 2025 20:08

pbrezina approved these changes Nov 4, 2025

View reviewed changes

pbrezina force-pushed the range_retrieval branch from 3b10082 to 3dc0a8d Compare November 4, 2025 11:48

pbrezina force-pushed the master branch 3 times, most recently from f5d64b3 to b854636 Compare November 4, 2025 14:27

alexey-tikhonov reviewed Nov 4, 2025

View reviewed changes

thalman force-pushed the range_retrieval branch from 3dc0a8d to 6d5df59 Compare November 5, 2025 10:46

pbrezina approved these changes Nov 5, 2025

View reviewed changes

thalman commented Nov 5, 2025

View reviewed changes

src/providers/ldap/sdap_async.c Show resolved Hide resolved

thalman force-pushed the range_retrieval branch from 6d5df59 to 28d7fec Compare November 5, 2025 14:13

thalman and others added 5 commits November 6, 2025 15:16

man: Update range retrieval information

46ffe03

sdap: replace objectClass string with SYSDB_OBJECTCLASS

6e637c7

Tests: Add range retrieval test for RHEL-75484

323ec0a

thalman force-pushed the range_retrieval branch from 28d7fec to 323ec0a Compare November 6, 2025 14:33

alexey-tikhonov added Changes requested and removed Waiting for review labels Nov 15, 2025

thalman marked this pull request as draft November 21, 2025 12:43

Range retrieval #8103

Are you sure you want to change the base?

Range retrieval #8103

Uh oh!

Conversation

thalman commented Sep 19, 2025

Uh oh!

thalman commented Sep 25, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alexey-tikhonov commented Oct 15, 2025

Uh oh!

Uh oh!

thalman commented Oct 21, 2025

Uh oh!

pbrezina left a comment

Choose a reason for hiding this comment

Uh oh!

pbrezina commented Nov 4, 2025

Uh oh!

alexey-tikhonov Nov 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

thalman Nov 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alexey-tikhonov Nov 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alexey-tikhonov Nov 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

thalman Nov 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alexey-tikhonov commented Nov 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pbrezina left a comment

Choose a reason for hiding this comment

Uh oh!

alexey-tikhonov Nov 4, 2025 •

edited

Loading

thalman Nov 10, 2025 •

edited

Loading

alexey-tikhonov Nov 10, 2025 •

edited

Loading

alexey-tikhonov Nov 4, 2025 •

edited

Loading

thalman Nov 10, 2025 •

edited

Loading

alexey-tikhonov commented Nov 4, 2025 •

edited

Loading