Warnings or reports rather than assertions

[smmaurer]

Currently orca_test uses an assertion model, where a failing check raises an exception and causes code execution to stop. @sablanchard has a use case where it would be better to print a report of all the failures at once. This is similar to issue #5.

(Raising an exception makes sense if you're using orca_test to check data dynamically within a simulation; printing a report makes sense for things like one-off data validation.)

Immediate solution

We realized you can get this behavior without changing orca_test, though, using an approach like this:

import orca_test
from orca_test import OrcaAssertionError

specs = [...]  # list OrcaSpecs here

problems = False
for spec in specs:
    try:
        orca_test.assert_orca_spec(spec)
    except OrcaAssertionError as e:
        problems = True
        print(str(e))
        pass

if problems:
    raise OrcaAssertionError("Problems found")

Here's a self-contained, fully-functional script demonstrating this usage: orca_test_demo.py

And this is what the output looks like:

Table 'households' is not registered
Table 'buildings' is already registered
Table 'badtable' is registered but cannot be generated
Column 'index' is not registered in table 'buildings'
Column 'price1' is already registered in table 'buildings'
Column 'badcol' is registered but cannot be generated
Column 'price1' is not set as the index of table 'buildings'
Column 'strings' has type 'object' (not numeric)
Column 'price2' is 20% missing, above limit of 0%
Column 'price1' has maximum value of 50, not 25
Column 'price1' has minimum value of -1, not 0
Column 'price2' is 20% missing, above limit of 10%
Column 'fkey_bad' has values that are not in 'zone_id'
Injectable 'nonexistent' is not registered
Injectable 'rate' is already registered
Injectable 'bad_inj' is registered but cannot be evaluated
Injectable 'dict' has type 'dict' (not numeric)
Injectable 'rate' has value of 0.64, less than 5
Injectable 'rate' has value of 0.56, greater than -5
Injectable 'rate' is not a dict
Injectable 'dict' does not have key 'Oakland'
Traceback (most recent call last):
  File "orca_test_demo.py", line 124, in <module>
    raise OrcaAssertionError("Problems found")
OrcaAssertionError: Problems found

Longer-term solution

Ultimately, though, we probably want to add multiple modes to orca_test, perhaps as global settings.

# Potential future functionality
orca_test.mode = 'warning'
orca_test.mode = 'report'