feat(newagent): Add a new Keyword Agent for pre-checking by rajuljha · Pull Request #109 · fossology/atarashi

rajuljha · 2025-07-02T07:02:00Z

Description

Introduced a new agent called KeywordAgent that performs pre-checks for license-possibility.

Keyword agent pre-checks for certain keywords for license-possibility detection.
Keywords are derived from:
- Nomos's STRINGS.in,
- FOSSology's licenseRef.json
- SPDX licenses
- SPDX exceptions

Changes

Add a new KeywordAgent class which handles KeywordAgent logic.
Modify licenseDownloader.py to include helper methods of downloading and building license_refs.csv file
Add building license_refs as a step in build_deps.py
Add license_keywords.txt and license_refs_combined.csv files
Add keyword_eval.py to evaluate KeywordAgent accuracy on NomosTestFiles

How to test

[!NOTE]
If building atarashi, run the build_depy.py script as well.

For running the atarashi agent:

chmod +x atarashi/atarashii.py
poetry run atarashi/atarashii.py -a <agent_name> <path_to_file_or_folder>

For running evaluator script:

chmod +x atarashi/evaluator/keyword_eval.py
poetry run atarashi/evaluator/keyword_eval.py

Screenshots

Evaluator results:

With keywordAgent being invoked:

Without KeywordAgent being invoked.

Kaushl2208

Hi @rajuljha,

I've reviewed the changes and recommended a few minor adjustments. Overall, the changes look good at first glance.

I have a question regarding the NomosTestFiles: Is it necessary to commit these files directly? Could we consider downloading them at runtime instead? For instance, if a user wishes to run the evaluator for Keywords, they could download the files when needed, allowing the evaluation to be performed dynamically.

From our perspective, the evaluation script serves as a form of functional testing, which we can utilize in CI stages to ensure everything is functioning correctly. In these scenarios, I believe we could download the TestFiles and execute the evaluator as required, correct?

Kaushl2208 · 2025-07-06T09:22:05Z

atarashi/agents/keywordAgent.py

+    if os.path.isfile(input_path):
+        results = agent.scan(input_path)
+        if results:
+            print(f"Scan results for {input_path}:")


Suggested change

print(f"Scan results for {input_path}:")

print(f"Keyword Scan results for {input_path}:")

Kaushl2208 · 2025-07-06T09:26:21Z

atarashi/agents/keywordAgent.py

+                file_path = os.path.join(root, file)
+                results = agent.scan(file_path)
+                if results:
+                    print(f"Scan results for {file_path}:")


Same here.

Since it is more of a support agent rather than being a License Detector as of now. We should also mention that specifically and I currently wondering if it can act as a standalone agent altogether or it can act as a bypasser for all the other agents? What do you think @rajuljha ?

KeywordAgent can act indepdantly as well by directly running the keyWordAgent.py file. Not through atarashi cli currently. Inside the atarashi cli, it only acts as a support agent for now. Let me know if it should be made accessible in the CLI as well.

Kaushl2208 · 2025-07-06T09:27:12Z

atarashi/agents/tfidf.py

+        warnings.simplefilter("ignore")
    sklearn_tfidf = TfidfVectorizer(min_df=1, use_idf=True, smooth_idf=True,
                                    sublinear_tf=True, tokenizer=tokenize,
+                                        token_pattern=None,


I am hoping this is for a bug fix??

Kaushl2208 · 2025-07-06T09:32:28Z

atarashi/atarashii.py

-    ngram_json = defaultJSON
+  # Validate compatibility between agent and similarity
+  if args.agent_name == "tfidf" and args.similarity not in ["CosineSim", "ScoreSim"]:
+    print("Error: TFIDF agent supports only CosineSim or ScoreSim.", file=sys.stderr)


Ideally this warning should be there in the help command for the agent right??

I have added this to the help command, but left this as a sanity check for the agent!

Kaushl2208 · 2025-07-06T09:34:22Z

atarashi/atarashii.py

-    result = {"file": os.path.abspath(inputPath), "results": result}
-    result = json.dumps(result, sort_keys=True, ensure_ascii=False, indent=4)
-    print(result + "\n")
+    keyword_ok = args.skip_keyword or keyword_scanner.scan(inputPath)


oww, My bad. Keyword agent actually working as a support agent here for any type of scan. :D

Kaushl2208 · 2025-07-06T09:38:59Z

atarashi/license/licenseDownloader.py

  :return: HTTP Pool Manager
  """
-  proxy_val = os.environ.get('http_proxy', False)
+  proxy_val = os.environ.get('http_proxy')


Good catch :P

Kaushl2208 · 2025-07-06T09:43:16Z

atarashi/license/licenseDownloader.py

+      cpuCount = os.cpu_count()
+
+      num_threads = threads if threads is not None else cpuCount
+      if num_threads is None:
+          num_threads = 1 # Fallback
+      if cpuCount is not None:
+          num_threads = min(num_threads, cpuCount * 2)


Can we make this a modular function??

rajuljha · 2025-07-06T14:02:48Z

Hi @rajuljha,

I've reviewed the changes and recommended a few minor adjustments. Overall, the changes look good at first glance.

I have a question regarding the NomosTestFiles: Is it necessary to commit these files directly? Could we consider downloading them at runtime instead? For instance, if a user wishes to run the evaluator for Keywords, they could download the files when needed, allowing the evaluation to be performed dynamically.

From our perspective, the evaluation script serves as a form of functional testing, which we can utilize in CI stages to ensure everything is functioning correctly. In these scenarios, I believe we could download the TestFiles and execute the evaluator as required, correct?

Yeah, we can do that. I'll take a look and fix this!

Keyword agent pre-checks for certain keywords for license-possibility detection. Keywords are from Nomos's STRINGS.in, FOSSology's license_ref.json and SPDX licenses and exceptions in a two stage pipeline. Signed-off-by: Rajul Jha <rajuljha49@gmail.com>

rajuljha mentioned this pull request Jul 2, 2025

Improving atarashi model fossology/fossology#3052

Closed

8 tasks

Kaushl2208 requested changes Jul 6, 2025

View reviewed changes

Kaushl2208 added the GSoC-2025 label Jul 16, 2025

rajuljha force-pushed the feat/newagent/Keyword branch from 6031e7a to adf3d98 Compare July 19, 2025 08:29

	print(f"Scan results for {input_path}:")
	print(f"Keyword Scan results for {input_path}:")

Conversation

rajuljha commented Jul 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Changes

How to test

Screenshots

Uh oh!

Kaushl2208 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rajuljha commented Jul 6, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

rajuljha commented Jul 2, 2025 •

edited

Loading