Skip to content

[BUG] Orchestrator integration test intermittently fails in Github offline tests #352

@oshoma

Description

@oshoma

Describe the bug
While running the github offline tests job for #344 I encountered a test failure in this file:

tests/integration_tests/test_orchestrator.py:75: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
sherpa_ai/orchestrator.py:54: in execute
    agent.run()
sherpa_ai/agents/base.py:92: in run
    action_output = self.act(action, inputs)
sherpa_ai/agents/base.py:141: in act
    return action.execute(**inputs)
sherpa_ai/actions/arxiv_search.py:38: in execute
    result = self.search_tool._run(query)
sherpa_ai/tools.py:58: in _run
    data = requests.get(url, timeout=HTTP_GET_TIMEOUT)

This is happening because I added a timeout to tools.py in #344 to make it safe to run in production. This test is doing an actual HTTP GET, which sometimes runs slowly and triggers the timeout.

The solution here is to mock the call to requests.get() in ArxivSearch._run at sherpa_ai/tools.py:58. We should not be using the network in our tests.

To Reproduce
Steps to reproduce the behavior:

  1. git checkout Fix bandit warnings #344
  2. make test
  3. If your computer runs the tests slowly, this test will fail

Alternatively, repeatedly run the offline tests job on github until it fails.

Or set the timeout value to a very small number to force the failure.

Expected behavior
The test should pass every time, and it should use a mock to prevent an actual network call via request.get().

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions