Add testing flag

[hdoupe]

I'm considering adding an environment variable to the automated tests that can be used to run a shorter/simpler version of the model. This would work like this:

$ CS_TEST=true py.test cs-config -v

Then models could check for the environment variable in their code:

def run_model(meta_param_dict, adjustment):
    if os.environ.get("CS_TEST") == "true":
        test_mode = True
    # ....

This would still give the model and C/S an opportunity to make sure that all required packages are installed and that outputs are formatted correctly without making model updates expensive in time or compute cost.

@jdebacker what do you think about this? (looking at you OG-USA...)

[jdebacker]

@hdoupe Something like this could be helpful for sure. Another option that @rickecon and I have been discussing is to have a timed test -- e.g., can we kick off a run and if things go for some predetermined amount of time (e.g., 10 min) without error, the test passes. We weren't sure how to implement this though.

[hdoupe]

Oh, good idea. What about something like this: https://stackoverflow.com/questions/492519/timeout-on-a-function-call

[jdebacker]

@hdoupe It looks like that suggestion would work. I have a simple idea for a test using that -- let me see if I can get it to work. Give me a day or two to get to it.

[hdoupe]

Sounds good @jdebacker.

I'm only thinking about it because automatic updates for models is a few days out and wanted to get in front of this issue.

[hdoupe]

@jdebacker is it possible to change the test run configuration to use the steady-state? Using the enhancement added in PR #14, you could do:

@pytest.mark.full_run
class TestFunctions1(CoreTestFunctions):
    get_version = functions.get_version
    get_inputs = functions.get_inputs
    validate_inputs = functions.validate_inputs
    run_model = functions.run_model
    # Only run steady state:
    ok_meta_param_dict = {"time_path": False}
    ok_adjustment = {"OG-USA Parameters": {"frisch": 0.41},
                     "Tax-Calculator Parameters": {}}
    bad_adjustment = {"OG-USA Parameters": {"frisch": 1.5},
                      "Tax-Calculator Parameters": {"STD": -1}}

This would make it possible to do a quick test to make sure that the outputs are formatted correctly, and then I'll trust that you have tested the code that handles the outputs generated from longer model runs.