Adds Robots.txt engine to dotnet device detection#720
Open
jwrosewell wants to merge 11 commits intomainfrom
Open
Adds Robots.txt engine to dotnet device detection#720jwrosewell wants to merge 11 commits intomainfrom
jwrosewell wants to merge 11 commits intomainfrom
Conversation
…n be shared with different types of engine implementation.
…concering the explicit setting of parallel for tests.
…lit out a) the query of the source engine, b) the construction of the data set, and c) the engine. The TAC and Native engines have been created separately to address #718 FEAT/MAJOR: Added robots.txt engine and tests to generate results only when query.robotstxt.[CrawlerUsage] is provided as evidence. Added model and services for generation of robots.txt to device detection package.
…d reference URIs.
…xed issue with Framework tests not passing due to version dependencies in version 8.
Contributor
Author
|
Note to reviewers: I'd like to re-read the submission before merging and following your review. |
…, namespace, test names
YaroslavVlasenko
approved these changes
Mar 30, 2026
Contributor
YaroslavVlasenko
left a comment
YaroslavVlasenko
left a comment
There was a problem hiding this comment.
Reviewed. Made minor fixes and pushed directly:
- Duplicate copyright line in TacEngine.cs
- Empty summary on PropertyKeyedDeviceEngineBaseBuilder constructor
- Pattern match variable not used in DataSetService.ConvertValues
- RobotsTxtData namespace mismatch (was PropertyKeyed.Data, now RobotsTxt.Data)
- Stale test names referencing old ConfigureForTac/ConfigureForNativeModel API
Closing #719 in favor of this PR.
justadreamer
requested changes
Mar 31, 2026
Contributor
justadreamer
left a comment
justadreamer
left a comment
There was a problem hiding this comment.
left some review comments, @jwrosewell please let us know if you have time to address - I can also ask @YaroslavVlasenko to fix and finalize the PR.
FiftyOne.DeviceDetection.PropertyKeyed/FlowElements/PropertyKeyedDeviceBaseEngine.cs
Outdated
Show resolved
Hide resolved
FiftyOne.DeviceDetection.PropertyKeyed/FlowElements/NativeEngine.cs
Outdated
Show resolved
Hide resolved
FiftyOne.DeviceDetection.RobotsTxt/Data/DevicePropertyMetaData.cs
Outdated
Show resolved
Hide resolved
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Adds a new engine called
RobotsTxtEnginewith associated builder to create robots.txt plan and annotated text files in responses whenquery.robotstxt.[usage]is provided in the evidence. Valid values are provided in theRobotsTxtEnginepublic static methods. Callers can use this engine to obtain a robots.txt entry ready to add to form a response. See tests for more details of parameters. Note: The engine only processes when there is evidence available. As such there can be situations where there is notRobotsTxtDatainstance available in the results.Refactored the
PropertyKeyedDeviceEngineand classes to make more flexible so that it can support the RobotsTxt requirement. Split out the TAC and Native engines from the single engine so that both can be used clearly in the same pipeline. Closes Remove TAC and Native code from base PropertyKeyedEngine #718Upgraded packages and fixed some tests and removed unused code.