Skip to content

Add python testing. #71

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
hsorby wants to merge 3 commits into
base: develop
Choose a base branch
from
Open

Add python testing. #71

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
c4d238e
Fix some compilation warnings.
hsorby Jul 31, 2018
ae33750
Add initial Python testing framework along with some documentation.
hsorby Jul 31, 2018
eead768
Rejig check_node_xyz_2d function to return a value. This is a more c…
hsorby Jul 31, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,3 +43,6 @@ CMakeLists.txt.user
*.pyc
__pycache__/

# PyCharm
.idea/

50 changes: 50 additions & 0 deletions cmake/PythonVirtualEnv.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@


set(VIRTUALENV_DIRECTORY "${CMAKE_CURRENT_BINARY_DIR}/venv_for_tests"
CACHE INTERNAL "Path to virtualenv" )
if (WIN32)
set(PYTHON_PLATFORM_BIN_DIR Scripts)
else ()
set(PYTHON_PLATFORM_BIN_DIR bin)
endif ()

find_program(VIRTUALENV_PYTHON_EXECUTABLE python PATHS "${VIRTUALENV_DIRECTORY}/${PYTHON_PLATFORM_BIN_DIR}" NO_DEFAULT_PATH)
mark_as_advanced(VIRTUALENV_PYTHON_EXECTUABLE)

if (NOT EXISTS "${VIRTUALENV_PYTHON_EXECTUABLE}")
function(_create_virtualenv_from_exec call)
execute_process(COMMAND
${call} --python=${PYTHON_EXECUTABLE} ${VIRTUALENV_DIRECTORY}
RESULT_VARIABLE RESULT
ERROR_VARIABLE ERROR
OUTPUT_VARIABLE OUTPUT
)
if(NOT "${RESULT}" STREQUAL "0")
message(STATUS "${RESULT}")
message(STATUS "${OUTPUT}")
message(STATUS "${ERROR}")
message(FATAL_ERROR "Could not create virtual environment.")
endif()
endfunction()

get_filename_component(_PYTHON_BIN "${PYTHON_EXECUTABLE}" PATH)
find_program(VIRTUALENV_EXECUTABLE virtualenv)
mark_as_advanced(VIRTUALENV_EXECUTABLE)

# Could also check for 'venv' and 'virtualenv' Python packages as we could use these instead of
# the virtualenv executable.
if (VIRTUALENV_EXECUTABLE)
message(STATUS "Creating virtual environment ...")
_create_virtualenv_from_exec("${VIRTUALENV_EXECUTABLE}")
else ()
message(FATAL_ERROR "Could not find virtualenv.")
endif ()

find_program(VIRTUALENV_PYTHON_EXECUTABLE python PATHS "${VIRTUALENV_DIRECTORY}/${PYTHON_PLATFORM_BIN_DIR}" NO_DEFAULT_PATH)

if (VIRTUALENV_PYTHON_EXECUTABLE)
message(STATUS "Creating virtual environment ... success")
else ()
message(FATAL_ERROR "Virtual environment Python executable does not exist.")
endif ()
endif ()
119 changes: 112 additions & 7 deletions documentation/testing.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,10 @@ Testing

Testing is an integral part of developing software and validating the code. It is also builds trust in users of the software that the software will work as expected. When adding new code to the library itself a test must also be added. Ideally the tests test every single line of code so that we have 100% code coverage. When we have good coverage of tests over the library we inherently get regression testing. We don't want to have new code changes breaking code that is already considered to be working.

For the testing of the Fortran code the pFUnit testing framework has been chosen. The pFUnit testing framework uses Python to manage some of the test generation, without Python we cannot build the tests.
For the testing of the Fortran code the pFUnit testing framework has been chosen. The pFUnit testing framework uses Python to manage some of the test generation, therefore without Python we cannot build the tests. For testing the code from Python we use the *unittest* testing module that is part of the Python distribution.

How to add a test
=================
How to add a Fortran test
=========================

All tests live under the *tests* tree and mirror what is in the source tree. In the following example we are going to add a new testing module for the *diagnostics* module in the *lib* directory from the *src* tree.

Expand All @@ -19,11 +19,11 @@ To start we are first going to make sure we have the correct structure that matc

tests/lib

exists and if not create it, from the command line on UNIX based oses this can be done with the *mkdir* command::
exists and if not create it, from the command line on UNIX based oses this can be done with the *mkdir* command (of course this should already be done, so there are big problems if you have to run this command!)::

mkdir tests/lib
mkdir -p tests/lib

Once the directory structure is correct we then create the testing module. Because we want to test the diagnostics module from the library we will create a test file named *test_diagnostics.pf* in the *tests/lib* directory. The *pf* extension indicates that this file is a hybrid Python fortran file, this file is a preprocessor input file which is Fortran free format file with preprocessor directives added. To create the test a Python script will generate a valid Fortran file from directives written into this file. With your favourite text editor create a file named *test_diagnostics.pf*. We could choose *vi* for this task as shown below but any text editor will work::
Once the directory structure is correct we then create the testing module. Because we want to test the *diagnostics* module from the library we will create a test file named *test_diagnostics.pf* in the *tests/lib* directory. The *pf* extension indicates that this file is a hybrid Python fortran file, this file is a preprocessor input file which is Fortran free format file with preprocessor directives added. To create the test a Python script will generate a valid Fortran file from directives written into this file. With your favourite text editor create a file named *test_diagnostics.pf*. We could choose *vi* for this task as shown below but any text editor will work::

vi tests/lib/test_diagnostics.pf

Expand All @@ -47,7 +47,6 @@ Into this file we will write our first test for the module. This test will chec

With our test written we now need to add this into the CMake build generation system.


Add test to CMake
-----------------

Expand Down Expand Up @@ -99,3 +98,109 @@ we will also execute all tests if we execute the command::

A handy flag to add to both of these commands is the *--verbose* flag. This gives us the details output from each test and not just the summary statement.


How to add a Python test
========================

In the following example we are going to add a new testing module for the *geometry* module for the *Python* bindings in the *src* tree.

Write test
----------

To start we are first going to make sure we have the correct structure that matches the *src* tree. Starting from the root directory of the lungsim repository we need to make sure that the directory::

tests/bindings/python

exists and if not create it, from the command line on UNIX based oses this can be done with the *mkdir* command (of course this should already be done, so there are big problems if you have to run this command!)::

mkdir -p tests/bindings/python

Once the directory structure is correct we then create the testing module. Because we want to test the *geometry* module from the library we will create a test file named *geometry_test.py* in the *tests/bindings/python* directory. We could choose *vi* for this task as shown below but any text editor will work::

vi tests/bindings/python/geometry_test.py

This file is going to be a standard Python file that makes use of the *unittest* unit testing framework. In this file we are going to write our first test for the module. This test will check that the *define_node_geometry_2d* method correctly sets the value of the nodes read from the *square.ipnode* file::

import os
import unittest

from aether.diagnostics import set_diagnostics_on
from aether.geometry import define_node_geometry_2d
from aether.arrays import check_node_xyz_2d

# Look to see if the 'TEST_RESOURCES_DIR' is set otherwise fallback to a path
# relative to this file.
if 'TEST_RESOURCES_DIR' in os.environ:
resources_dir = os.environ['TEST_RESOURCES_DIR']
else:
here = os.path.abspath(os.path.dirname(__file__))
resources_dir = os.path.join(here, 'resources')


class GeometryTestCase(unittest.TestCase):

def test_read_square(self):
set_diagnostics_on(False)
define_node_geometry_2d(os.path.join(resources_dir, 'square.ipnode'))
value = check_node_xyz_2d(1, 1, 10)
self.assertEqual(10, value)


if __name__ == '__main__':
unittest.main()


The first thing to note is that when we are handling external resources like files we should be explicit in where they are coming from. This is reason for the following statement::

if 'TEST_RESOURCES_DIR' in os.environ:
resources_dir = os.environ['TEST_RESOURCES_DIR']
else:
here = os.path.abspath(os.path.dirname(__file__))
resources_dir = os.path.join(here, 'resources')

When we are using ctest to run the tests the *TEST_RESOURCES_DIR* environment variable defines the location of the resources directory. If this environment variable is not found then the fallback is to set the resources directory relative to the file itself. This allows us to run the test file in different environments. The end result is that we can explicitly state (in a relative sense) the location of the *square.ipnode* file resource for this test.

The second point to note is that our test is defined within a **class** that derives from *unittest.TestCase*. Any class deriving from *unittest.TestCase* found in this file will be run as a test. This means we are free to have multiple classes derived from *unittest.testcase* if we so choose. The benefit of this is that we can group our tests by some heuristic.

The third point is on the test itself. In this test we are testing to make sure that the correct value for the node is set when reading in a node geometry file. We use the *unittest* framework to assert that the value of the node is indeed 10.

The final point is about the last two lines of the file. It is these two lines that get executed by ctest if they are missing the test will actually pass this is because no tests will have been run therefor it is important that these two lines are present at the bottom of every file that defines classes derived from *unittest.TestCase*.

With our test written we now need to add this into the CMake build generation system.

Add test to CMake
-----------------

To make CMake aware of the new test we need to add an new entry in to the *TEST_SRCS* CMake variable in the file *tests/bindings/python/CMakeLists.txt*. For this example we need to add *geometry_test.py*. With this being our first Python test our *TEST_SRCS* variable will look like the following::

set(TEST_SRCS
geometry_test.py
)

Over time we should have a list of test files defined here.

Run test
--------

The Python tests do not need building, as such, they do however require a little preprocessing for ctest to run them. We can make sure the preprocessing is done in one of two ways.

1. Execute the test command::

make test

2. Make CMake perform the preprocessing, and then run the tests with ctest::

cmake .
ctest

All of these commands must of course be executed from within the build directory.

We can of course run just some of the tests using the *-R* flag to ctest. To run just the Python tests we could execute the following command::

ctest -R python_

Don't forget about the verbose flag::

ctest -R python_ -V

This command will show us more detailed output from the Python tests.
1 change: 0 additions & 1 deletion src/bindings/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,6 @@ mark_as_advanced(SWIG_EXECUTABLE)
if(SWIG_FOUND)
option(AETHER_BUILD_PYTHON_BINDINGS "Build Python bindings for ${PROJECT_NAME}" YES)


if(AETHER_BUILD_PYTHON_BINDINGS)
find_package(PythonInterp)
find_package(PythonLibs)
Expand Down
7 changes: 7 additions & 0 deletions src/bindings/c/arrays.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,3 +7,10 @@ void set_node_field_value(int row, int col, double value)
{
set_node_field_value_c(&row, &col, &value);
}

double check_node_xyz_2d(int row, int col)
{
double value;
check_node_xyz_2d_c(&row, &col, &value);
return value;
}
1 change: 1 addition & 0 deletions src/bindings/c/arrays.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,5 +5,6 @@
#include "symbol_export.h"

SHO_PUBLIC void set_node_field_value(int row, int col, double value);
SHO_PUBLIC double check_node_xyz_2d(int row, int col);
Copy link
Contributor Author

@hsorby hsorby Jul 31, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function should probably be called get_node_xyz_2d.


#endif /* AETHER_ARRAYS_H */
1 change: 1 addition & 0 deletions src/bindings/c/exports.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ void export_node_geometry_2d_c(const char *EXNODEFILE, int *EXNODEFILE_LEN, cons
void export_data_geometry_c(const char *EXDATAFILE, int *EXDATAFILE_LEN, const char *name, int *name_len, int *offset);
void export_elem_field_c(const char *EXELEMFIELD, int *EXELEMFIELD_LEN,
const char *name, int *name_len, const char *field_name, int *field_name_len);
void export_terminal_ssgexch_c(const char *EXNODEFILE, int *EXNODEFILE_LEN, const char *name, int *name_len);

void export_1d_elem_field(int ne_field, const char *EXELEMFILE, const char *group_name, const char *field_name )
{
Expand Down
2 changes: 1 addition & 1 deletion src/bindings/c/geometry.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,7 @@ int get_local_node_f(const char *ndimension, const char *np_global)
{
int dimension_len = strlen(ndimension);
int np_global_len = strlen(np_global);
get_local_node_f_c(ndimension, &dimension_len, np_global, &np_global_len);
return get_local_node_f_c(ndimension, &dimension_len, np_global, &np_global_len);
}

void define_rad_from_geom(const char *order_system, double control_param, const char *start_from,
Expand Down
2 changes: 2 additions & 0 deletions src/bindings/c/pressure_resistance_flow.c
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
#include "pressure_resistance_flow.h"

#include <string.h>

void evaluate_prq_c(const char *mesh_type, int *mesh_type_len, int *grav_dirn, double *grav_factor, const char *bc_type, int *bc_type_len, double *inlet_bc, double *outlet_bc);

void evaluate_prq(const char *mesh_type, int grav_dirn, double grav_factor, const char *bc_type, double inlet_bc, double outlet_bc)
Expand Down
1 change: 1 addition & 0 deletions tests/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,4 @@ add_subdirectory(pFUnit-3.2.9)
include("${CMAKE_CURRENT_BINARY_DIR}/pFUnit-3.2.9/pFUnitConfig.cmake")

add_subdirectory(lib)
add_subdirectory(bindings)
3 changes: 3 additions & 0 deletions tests/bindings/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@

add_subdirectory(python)

25 changes: 25 additions & 0 deletions tests/bindings/python/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@

# Create a Python virtual environment to use for testing. This is only done once.
include(PythonVirtualEnv)

set(TEST_SRCS
geometry_test.py
)

foreach(_TEST ${TEST_SRCS})
# We'll actually take a copy of the test and run it from the build directory and not the source directory.
configure_file(${_TEST} ${_TEST} COPYONLY)

# Set up the test.
get_filename_component(_TEST_NAME ${_TEST} NAME_WE)
set(_TEST_NAME python_${_TEST_NAME})
add_test(NAME ${_TEST_NAME} COMMAND ${VIRTUALENV_PYTHON_EXECUTABLE} ${_TEST})
if (WIN32)
set_tests_properties(${_TEST_NAME} PROPERTIES
ENVIRONMENT "PATH=$<TARGET_FILE_DIR:aether>\;%PATH%;PYTHONPATH=${PROJECT_BINARY_DIR}/src/bindings/python;TEST_RESOURCES_DIR=${CMAKE_CURRENT_SOURCE_DIR}/resources")
else ()
set_tests_properties(${_TEST_NAME} PROPERTIES
ENVIRONMENT "PYTHONPATH=${PROJECT_BINARY_DIR}/src/bindings/python;TEST_RESOURCES_DIR=${CMAKE_CURRENT_SOURCE_DIR}/resources")
endif ()

endforeach()
27 changes: 27 additions & 0 deletions tests/bindings/python/geometry_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
import os
import unittest

from aether.diagnostics import set_diagnostics_on
from aether.geometry import define_node_geometry_2d
from aether.arrays import check_node_xyz_2d

# Look to see if the 'TEST_RESOURCES_DIR' is set otherwise fallback to a path
# relative to this file.
if 'TEST_RESOURCES_DIR' in os.environ:
resources_dir = os.environ['TEST_RESOURCES_DIR']
else:
here = os.path.abspath(os.path.dirname(__file__))
resources_dir = os.path.join(here, 'resources')


class GeometryTestCase(unittest.TestCase):

def test_read_square(self):
set_diagnostics_on(False)
define_node_geometry_2d(os.path.join(resources_dir, 'square.ipnode'))
value = check_node_xyz_2d(1, 1)
self.assertEqual(100, value)


if __name__ == '__main__':
unittest.main()
68 changes: 68 additions & 0 deletions tests/bindings/python/resources/square.ipnode
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
CMISS Version 1.21 ipnode File Version 2
Heading: 52

The number of nodes is [ 35]: 4
Number of coordinates [ 3]: 3
Do you want prompting for different versions of nj=1 [N]? N
Do you want prompting for different versions of nj=2 [N]? N
Do you want prompting for different versions of nj=3 [N]? N
The number of derivatives for coordinate 1 is [0]: 3
The number of derivatives for coordinate 2 is [0]: 3
The number of derivatives for coordinate 3 is [0]: 3

Node number [ 52]: 52
The Xj(1) coordinate is [ 0.00000E+00]: 1.00000000e+02
The derivative wrt direction 1 is [ 0.00000E+00]: 1.0000000000e+00
The derivative wrt direction 2 is [ 0.00000E+00]: 0.0000000000e+00
The derivative wrt directions 1 & 2 is [ 0.00000E+00]: 0.0000000000e+00
The Xj(2) coordinate is [ 0.00000E+00]: 0.000000000e+00
The derivative wrt direction 1 is [ 0.00000E+00]: 0.00000000e+00
The derivative wrt direction 2 is [ 0.00000E+00]: -1.00000000e+00
The derivative wrt directions 1 & 2 is [ 0.00000E+00]: 0.0000000000e+00
The Xj(3) coordinate is [ 0.00000E+00]: 0.00000000e+00
The derivative wrt direction 1 is [ 0.00000E+00]: 0.000000001e+00
The derivative wrt direction 2 is [ 0.00000E+00]: 0.000000000e+00
The derivative wrt directions 1 & 2 is [ 0.00000E+00]: 0.000000000e+00

Node number [ 53]: 53
The Xj(1) coordinate is [ 0.00000E+00]: 2.00000000000e+02
The derivative wrt direction 1 is [ 0.00000E+00]: 1.000000000e+00
The derivative wrt direction 2 is [ 0.00000E+00]: 0.000000000e-00
The derivative wrt directions 1 & 2 is [ 0.00000E+00]: 0.000000000e+00
The Xj(2) coordinate is [ 0.00000E+00]: 0.000000000e+00
The derivative wrt direction 1 is [ 0.00000E+00]: 0.000000000e+00
The derivative wrt direction 2 is [ 0.00000E+00]: -1.000000000e+00
The derivative wrt directions 1 & 2 is [ 0.00000E+00]: 0.000000000e+00
The Xj(3) coordinate is [ 0.00000E+00]: 0.000000000e+00
The derivative wrt direction 1 is [ 0.00000E+00]: 0.000000000e+00
The derivative wrt direction 2 is [ 0.00000E+00]: 0.000000000e+00
The derivative wrt directions 1 & 2 is [ 0.00000E+00]: 0.000000000e+00

Node number [ 54]: 54
The Xj(1) coordinate is [ 0.00000E+00]: 1.000000000e+02
The derivative wrt direction 1 is [ 0.00000E+00]: 1.0000000000e+00
The derivative wrt direction 2 is [ 0.00000E+00]: 0.0000000000e+00
The derivative wrt directions 1 & 2 is [ 0.00000E+00]: 0.0000000000e+00
The Xj(2) coordinate is [ 0.00000E+00]: -1.00000000e+02
The derivative wrt direction 1 is [ 0.00000E+00]: 0.00000000000e+00
The derivative wrt direction 2 is [ 0.00000E+00]: -1.00000000000e+00
The derivative wrt directions 1 & 2 is [ 0.00000E+00]: 0.00000000000e+00
The Xj(3) coordinate is [ 0.00000E+00]: 0.000000000e+00
The derivative wrt direction 1 is [ 0.00000E+00]: 0.0000000000e+00
The derivative wrt direction 2 is [ 0.00000E+00]: 0.0000000000e+00
The derivative wrt directions 1 & 2 is [ 0.00000E+00]: 0.0000000000e+00

Node number [ 56]: 56
The Xj(1) coordinate is [ 0.00000E+00]: 2.00000000e+02
The derivative wrt direction 1 is [ 0.00000E+00]: 1.000000000e+00
The derivative wrt direction 2 is [ 0.00000E+00]: 0.000000000e+00
The derivative wrt directions 1 & 2 is [ 0.00000E+00]: 0.000000000e+00
The Xj(2) coordinate is [ 0.00000E+00]: -1.00000000e+02
The derivative wrt direction 1 is [ 0.00000E+00]: 0.000000000e+00
The derivative wrt direction 2 is [ 0.00000E+00]: -1.000000001e+00
The derivative wrt directions 1 & 2 is [ 0.00000E+00]: 0.000000000e+00
The Xj(3) coordinate is [ 0.00000E+00]: 0.00000000e+00
The derivative wrt direction 1 is [ 0.00000E+00]: 0.0000000e+00
The derivative wrt direction 2 is [ 0.00000E+00]: 0.0000000e+00
The derivative wrt directions 1 & 2 is [ 0.00000E+00]: 0.0000000e+00