Added ACL feature #2190

AlexBlueSteele · 2025-10-08T20:54:58Z

Ticket: #2186
Test run output:
File index

{
  "_index": "test",
  "_id": "REDACTED",
  "_score": null,
  "_source": {
    "meta": {
      "raw": {
        "X-TIKA:Parsed-By": "org.apache.tika.parser.DefaultParser",
        "X-TIKA:Parsed-By-Full-Set": "org.apache.tika.parser.DefaultParser",
        "resourceName": "test.xml",
        "Content-Type": "application/xml"
      }
    },
    "file": {
      "extension": "xml",
      "content_type": "application/xml",
      "created": "REDACTED",
      "last_modified": "REDACTED",
      "last_accessed": "REDACTED",
      "indexing_date": "REDACTED",
      "filesize": REDACTED,
      "filename": "test.xml",
      "url": "file://REDACTED//test.xml",
      "indexed_chars": REDACTED
    },
    "path": {
      "root": "REDACTED",
      "virtual": "\\REDACTED\\test.xml",
      "real": "\\\\REDACTED\\test.xml"
    },
    "attributes": {
      "owner": "DOMAIN1\\testuser",
      "permissions": 0,
      "acl": [
        {
          "principal": "DOMAIN1\\testuser",
          "type": "ALLOW",
          "permissions": [
            "APPEND_DATA",
            "DELETE",
            "DELETE_CHILD",
            "EXECUTE",
            "READ_ACL",
            "READ_ATTRIBUTES",
            "READ_DATA",
            "READ_NAMED_ATTRS",
            "SYNCHRONIZE",
            "WRITE_ACL",
            "WRITE_ATTRIBUTES",
            "WRITE_DATA",
            "WRITE_NAMED_ATTRS",
            "WRITE_OWNER"
          ]
        },
        {
          "principal": "DOMAIN1\\Test Admins",
          "type": "ALLOW",
          "permissions": [
            "APPEND_DATA",
            "DELETE",
            "DELETE_CHILD",
            "EXECUTE",
            "READ_ACL",
            "READ_ATTRIBUTES",
            "READ_DATA",
            "READ_NAMED_ATTRS",
            "SYNCHRONIZE",
            "WRITE_ACL",
            "WRITE_ATTRIBUTES",
            "WRITE_DATA",
            "WRITE_NAMED_ATTRS",
            "WRITE_OWNER"
          ]
		}
    }
  },
  "highlight": {
    "path.virtual": [
      "<b>\\REDACTED\\test.xml</b>"
    ],
    "file.filename": [
      "<b>test.xml</b>"
    ]
  },
  "sort": [
    REDACTED
  ]
}

Folder index

    {
        "path": {
            "root": "REDATED",
            "virtual": "Group3Folder",
            "real": "W:\\Group3Folder"
        },
        "file": {
            "content_type": "text/directory",
            "created": "REDACTED",
            "last_modified": "REDACTED",
            "last_accessed": "REDACTED",
            "filename": "Group3Folder"
        },
        "attributes": {
            "owner": "DOMAIN\\FileShareManagers",
            "permissions": 0,
            "acl": [
                {
                    "principal": "DOMAIN\\group3",
                    "type": "ALLOW",
                    "permissions": [
                        "APPEND_DATA",
                        "DELETE",
                        "DELETE_CHILD",
                        "EXECUTE",
                        "READ_ACL",
                        "READ_ATTRIBUTES",
                        "READ_DATA",
                        "READ_NAMED_ATTRS",
                        "SYNCHRONIZE",
                        "WRITE_ACL",
                        "WRITE_ATTRIBUTES",
                        "WRITE_DATA",
                        "WRITE_NAMED_ATTRS",
                        "WRITE_OWNER"
                    ],
                    "flags": [
                        "DIRECTORY_INHERIT",
                        "FILE_INHERIT",
                        "INHERIT_ONLY"
                    ]
                },
                {
                    "principal": "BUILTIN\\Administrators",
                    "type": "ALLOW",
                    "permissions": [
                        "APPEND_DATA",
                        "DELETE",
                        "DELETE_CHILD",
                        "EXECUTE",
                        "READ_ACL",
                        "READ_ATTRIBUTES",
                        "READ_DATA",
                        "READ_NAMED_ATTRS",
                        "SYNCHRONIZE",
                        "WRITE_ACL",
                        "WRITE_ATTRIBUTES",
                        "WRITE_DATA",
                        "WRITE_NAMED_ATTRS",
                        "WRITE_OWNER"
                    ],
                    "flags": [
                        "DIRECTORY_INHERIT",
                        "FILE_INHERIT"
                    ]
                },
                {
                    "principal": "DOMAIN\\Domain Users",
                    "type": "ALLOW",
                    "permissions": [
                        "EXECUTE",
                        "READ_ACL",
                        "READ_ATTRIBUTES",
                        "READ_DATA",
                        "READ_NAMED_ATTRS",
                        "SYNCHRONIZE"
                    ]
                },
                {
                    "principal": "DOMAIN\\service accounts",
                    "type": "ALLOW",
                    "permissions": [
                        "APPEND_DATA",
                        "DELETE",
                        "DELETE_CHILD",
                        "EXECUTE",
                        "READ_ACL",
                        "READ_ATTRIBUTES",
                        "READ_DATA",
                        "READ_NAMED_ATTRS",
                        "SYNCHRONIZE",
                        "WRITE_ACL",
                        "WRITE_ATTRIBUTES",
                        "WRITE_DATA",
                        "WRITE_NAMED_ATTRS",
                        "WRITE_OWNER"
                    ],
                    "flags": [
                        "DIRECTORY_INHERIT",
                        "FILE_INHERIT"
                    ]
                },
                {
                    "principal": "DOMAIN\\FileShareManagers",
                    "type": "ALLOW",
                    "permissions": [
                        "APPEND_DATA",
                        "DELETE",
                        "DELETE_CHILD",
                        "EXECUTE",
                        "READ_ACL",
                        "READ_ATTRIBUTES",
                        "READ_DATA",
                        "READ_NAMED_ATTRS",
                        "SYNCHRONIZE",
                        "WRITE_ACL",
                        "WRITE_ATTRIBUTES",
                        "WRITE_DATA",
                        "WRITE_NAMED_ATTRS",
                        "WRITE_OWNER"
                    ]
                },
                {
                    "principal": "\\CREATOR OWNER",
                    "type": "ALLOW",
                    "permissions": [
                        "APPEND_DATA",
                        "DELETE",
                        "DELETE_CHILD",
                        "EXECUTE",
                        "READ_ACL",
                        "READ_ATTRIBUTES",
                        "READ_DATA",
                        "READ_NAMED_ATTRS",
                        "SYNCHRONIZE",
                        "WRITE_ACL",
                        "WRITE_ATTRIBUTES",
                        "WRITE_DATA",
                        "WRITE_NAMED_ATTRS",
                        "WRITE_OWNER"
                    ],
                    "flags": [
                        "DIRECTORY_INHERIT",
                        "FILE_INHERIT",
                        "INHERIT_ONLY"
                    ]
                },
                {
                    "principal": "NT AUTHORITY\\SYSTEM",
                    "type": "ALLOW",
                    "permissions": [
                        "APPEND_DATA",
                        "DELETE",
                        "DELETE_CHILD",
                        "EXECUTE",
                        "READ_ACL",
                        "READ_ATTRIBUTES",
                        "READ_DATA",
                        "READ_NAMED_ATTRS",
                        "SYNCHRONIZE",
                        "WRITE_ACL",
                        "WRITE_ATTRIBUTES",
                        "WRITE_DATA",
                        "WRITE_NAMED_ATTRS",
                        "WRITE_OWNER"
                    ],
                    "flags": [
                        "DIRECTORY_INHERIT",
                        "FILE_INHERIT"
                    ]
                },
                {
                    "principal": "DOMAIN\\FileShareManagers",
                    "type": "ALLOW",
                    "permissions": [
                        "APPEND_DATA",
                        "DELETE",
                        "DELETE_CHILD",
                        "EXECUTE",
                        "READ_ACL",
                        "READ_ATTRIBUTES",
                        "READ_DATA",
                        "READ_NAMED_ATTRS",
                        "SYNCHRONIZE",
                        "WRITE_ACL",
                        "WRITE_ATTRIBUTES",
                        "WRITE_DATA",
                        "WRITE_NAMED_ATTRS",
                        "WRITE_OWNER"
                    ],
                    "flags": [
                        "DIRECTORY_INHERIT",
                        "FILE_INHERIT",
                        "INHERIT_ONLY"
                    ]
                }
            ]
        },
        "_id": "REDACTED"
    },

Note

Adds fs.acl_support to collect and index file/folder ACLs, updates mappings/templates, wiring, docs, and tests.

Framework/Core:
- Introduces FileAcl and FsCrawlerUtil#getFileAcls(...); adds ACL extraction for local files.
- Extends beans.Attributes with acl and beans.Folder with attributes; propagate attributes/ACLs in FsParserAbstract.
- FileAbstractModel carries acls; FileAbstractorFile fills ACLs (FTP/SSH return empty lists).
Settings:
- New flag fs.acl_support (+ defaults, parsing, validation warning if attributes_support is false).
Elasticsearch:
- Add component template fscrawler_mapping_attributes with attributes.acl.{principal,type,permissions,flags}.
- Include attributes component in folders index template; load it at startup.
Docs:
- Document acl_support in admin guides (index.rst, local-fs.rst, rest.rst) with examples.
Tests:
- Add ACL-related tests (FsCrawlerUtilTest, JsonUtilTest, settings loader/parser tests) and update sample configs.

^{Written by Cursor Bugbot for commit b61843c. This will update automatically on new commits. Configure here.}

dadoonet · 2025-10-22T11:00:37Z

@AlexBlueSteele Do you want to continue on this PR? If so, could you make sure that all the tests are passing locally?

And then I can start reviewing the code ;)

AlexBlueSteele · 2025-11-06T16:02:34Z

@AlexBlueSteele Do you want to continue on this PR? If so, could you make sure that all the tests are passing locally?

And then I can start reviewing the code ;)

I will work on this. Thanks!

sonarqubecloud · 2025-11-06T17:02:06Z

Quality Gate passed

Issues
7 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

dadoonet · 2025-11-06T20:39:34Z

I'm wondering if we need the acl_support setting? I believe that the other one is enough. WDYT?

AlexBlueSteele · 2025-11-07T17:05:16Z

I'm wondering if we need the acl_support setting? I believe that the other one is enough. WDYT?

I think it's nice to have because some people might not care about ACLs when crawling on windows. It adds slight overhead in storage and performance.

AlexBlueSteele · 2025-11-07T17:15:42Z

@dadoonet I updated the demo data above to reflect the ACL collection for the folder index as well. The final part that I think needs to be changed its the verbosity of the debug statements. I don't know how verbose you want it. Thoughts?

dadoonet

That's a great start and thank you so much for working on this.
Bonus point for writing some documentation ;)

Could you fix the issues I mentioned before?

Also, next steps would be IMO to add if possible an Integration Tests about this. May be in FsCrawlerTestAttributesIT?

Would that be possible for you?

Also did you run the code locally and does it do what you are expecting?

I'm still wondering if we really need another acl_support attribute or consider that we want want to collect all the available metadata on files and folders whenever we are asking for attributes...

core/src/main/java/fr/pilato/elasticsearch/crawler/fs/FsParserAbstract.java

...ler-abstract/src/main/java/fr/pilato/elasticsearch/crawler/fs/crawler/FileAbstractModel.java

core/src/main/java/fr/pilato/elasticsearch/crawler/fs/FsParserAbstract.java

beans/src/main/java/fr/pilato/elasticsearch/crawler/fs/beans/Folder.java

framework/src/main/java/fr/pilato/elasticsearch/crawler/fs/framework/FsCrawlerUtil.java

framework/src/test/java/fr/pilato/elasticsearch/crawler/fs/framework/FsCrawlerUtilTest.java

AlexBlueSteele · 2025-11-25T16:34:21Z

That's a great start and thank you so much for working on this. Bonus point for writing some documentation ;)

Could you fix the issues I mentioned before?

Also, next steps would be IMO to add if possible an Integration Tests about this. May be in FsCrawlerTestAttributesIT?

Would that be possible for you?

Also did you run the code locally and does it do what you are expecting?

I'm still wondering if we really need another acl_support attribute or consider that we want want to collect all the available metadata on files and folders whenever we are asking for attributes...

I will work on this as soon as I get the chance! Thanks for spending the time to CR!!

sonarqubecloud · 2025-12-19T02:10:23Z

Quality Gate passed

Issues
5 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

AlexBlueSteele added 3 commits October 7, 2025 16:18

Added ACL

0650126

Added ACL flag

d3f536e

Added debug

4c235fa

AlexBlueSteele requested a review from dadoonet as a code owner October 8, 2025 20:54

Fixed output

472440b

AlexBlueSteele added 2 commits November 6, 2025 10:55

Bumped to new release version

63201fe

Merged in debug change

18a9feb

AlexBlueSteele marked this pull request as draft November 6, 2025 16:03

AlexBlueSteele added 3 commits November 6, 2025 11:20

Added ACLs for folders

e1f132e

Added to pass build tests

124b0f5

Added to pass IT

b61843c

dadoonet marked this pull request as ready for review November 18, 2025 15:21

dadoonet requested changes Nov 18, 2025

View reviewed changes

Added changes from code review

47ded9a

Added ACL feature #2190

Are you sure you want to change the base?

Added ACL feature #2190

Conversation

AlexBlueSteele commented Oct 8, 2025 • edited by cursor bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dadoonet commented Oct 22, 2025

Uh oh!

AlexBlueSteele commented Nov 6, 2025

Uh oh!

sonarqubecloud bot commented Nov 6, 2025

Quality Gate passed

Uh oh!

dadoonet commented Nov 6, 2025

Uh oh!

AlexBlueSteele commented Nov 7, 2025

Uh oh!

AlexBlueSteele commented Nov 7, 2025

Uh oh!

dadoonet left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AlexBlueSteele commented Nov 25, 2025

Uh oh!

sonarqubecloud bot commented Dec 19, 2025

Quality Gate passed

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

AlexBlueSteele commented Oct 8, 2025 •

edited by cursor bot

Loading