Skip to content

Replace pFUnit with a simpler testing framework. #171

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
hsorby wants to merge 3 commits into
base: develop
Choose a base branch
from
Open

Replace pFUnit with a simpler testing framework. #171

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
dd031e7
Remove pFUnit-3.2.9.
hsorby Dec 2, 2024
213bd0f
Replace pFUnit with a simpler testing framework.
hsorby Dec 2, 2024
7350b32
Add documentation for new testing framework.
hsorby Dec 2, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@


cmake_minimum_required(VERSION 3.13)
cmake_minimum_required(VERSION 3.18)
project(Aether VERSION 0.1.0 LANGUAGES Fortran C CXX)

# Policy settings
Expand Down
178 changes: 115 additions & 63 deletions documentation/testing.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,108 +15,160 @@ The pFUnit testing framework uses Python to manage some of the test generation,
How to add a test
=================

All tests live under the *tests* tree and mirror what is in the source tree.
In the following example we are going to add a new testing module for the *diagnostics* module in the *lib* directory from the *src* tree.
All tests live under the *tests* directory.
In the following example we are going to add a new testing module for the *diagnostics* module which resides in the *src/lib* directory.

Write test
----------

To start we are first going to make sure we have the correct structure that matches the *src* tree.
Starting from the root directory of the lungsim repository we need to make sure that the directory::
To start we create the testing module.
Because we want to test the diagnostics module from the library we will create a test file named *test_diagnostics.f90* in the *tests* directory.
With your favourite text editor create a file named *test_diagnostics.f90*.
We could choose *vi* for this task as shown below but any text editor will work::

tests/lib
vi tests/test_diagnostics.f90

exists and if not create it, from the command line on UNIX based oses this can be done with the *mkdir* command::
Into this file we will write our test for the module.
First, we are going to add declare a module called *test_diagnostics*::

mkdir tests/lib
module test_diagnostics

Once the directory structure is correct we then create the testing module.
Because we want to test the diagnostics module from the library we will create a test file named *test_diagnostics.pf* in the *tests/lib* directory.
The *pf* extension indicates that this file is a hybrid Python fortran file, this file is a preprocessor input file which is Fortran free format file with preprocessor directives added.
To create the test a Python script will generate a valid Fortran file from directives written into this file.
With your favourite text editor create a file named *test_diagnostics.pf*.
We could choose *vi* for this task as shown below but any text editor will work::
!> We will add framework declarations and code here.

vi tests/lib/test_diagnostics.pf
!> Then we will write our test(s) here.

Into this file we will write our first test for the module.
This test will check that the diagnositcs flag has been set when using the *set_diagnostics_on* subroutine::
end module test_diagnostics

@test
subroutine testSetDiagnostics()
use pfunit_mod
use diagnostics, only: get_diagnostics_on, set_diagnostics_on
implicit none
Into this module, we have to add some boiler plate code to declare our test to the framework and make it findable::

logical :: state
use testdrive, only : new_unittest, unittest_type, error_type, check
implicit none
private

call get_diagnostics_on(state)
@assertFalse(state)
call set_diagnostics_on(.true.)
call get_diagnostics_on(state)
@assertTrue(state)
In this chunk of code we can see that we are using the *new_unittest, unittest_type, error_type, check* subroutines from the *testdrive* module.
We will also declare a helper subroutine to provide a list of test names to the framework::

end subroutine testSetDiagnostics
public :: collect_diagnostics

With our test written we now need to add this into the CMake build generation system.
Now we can start with defining the *collect_diagnostics* subroutine and our test(s).
We are going to call our test *test_set_and_get*, and this is what our *collect_diagnostics* subroutine will report to the framework::

!> Collect all exported unit tests
subroutine collect_diagnostics(testsuite)
!> Collection of tests
type(unittest_type), allocatable, intent(out) :: testsuite(:)

Add test to CMake
-----------------
testsuite = [ &
new_unittest("test_set_and_get", test_set_and_get) &
]

end subroutine collect_diagnostics

Next we define the *test_set_and_get* test::

subroutine test_set_and_get(error)
use diagnostics, only: get_diagnostics_on, set_diagnostics_on
implicit none

type(error_type), allocatable, intent(out) :: error

logical :: level

call get_diagnostics_on(level)
call check(error, .false., level)
if (allocated(error)) return

call set_diagnostics_on(.true.)
call get_diagnostics_on(level)
call check(error, .true., level)
if (allocated(error)) return

end subroutine test_set_and_get

The complete *tests_diagnostics.f90* file looks like this::

module test_diagnostics
use testdrive, only : new_unittest, unittest_type, error_type, check
implicit none
private

public :: collect_diagnostics

The first task to do when adding a test to the CMake files is to check that a CMake file exists.
When adding a test to a new directory, as we are doing here, there won't be a CMake file for us to use.
To fix this we first need to tell CMake that a new subdirectory is available.
We do this by adding a *sub_directory* command into an existing *CMakeLists.txt* file in a parent directory of the directory we have just added a test to.
In our example we would edit the file (any text editor will do, don't feel you need to use *vi*)::
contains

vi tests/CMakeLists.txt
!> Collect all exported unit tests
subroutine collect_diagnostics(testsuite)
!> Collection of tests
type(unittest_type), allocatable, intent(out) :: testsuite(:)

and add the line at the bottom of the file::
testsuite = [ &
new_unittest("test_set_and_get", test_set_and_get) &
]

add_subdirectory(lib)
end subroutine collect_diagnostics

Then we need to create a new *CMakeLists.txt* (the capitalisation of this file is important) file in the *tests/lib* directory (any text editor will do, don't feel you need to use *vi*)::
subroutine test_set_and_get(error)
use diagnostics, only: get_diagnostics_on, set_diagnostics_on
implicit none

vi tests/lib/CMakeLists.txt
type(error_type), allocatable, intent(out) :: error

and add the following to create an executable test that will work with CTest (we will also be able to execute this test directly)::
logical :: level

# Add all the files that make a single test, we could have multiple files testing
# the same module. Don't add test files into the same test that test different modules.
# These are all .pf files.
set(DIAGNOSTICS_TEST_SRCS
test_diagnostics.pf)
call get_diagnostics_on(level)
call check(error, .false., level)
if (allocated(error)) return

# Make use of the pFUnit helper function to create a test.
# Arguments : - test_package_name: Name of the test package
# - test_sources : List of pf-files to be compiled
# - extra_sources : List of extra Fortran source code used for testing (if none, input empty string "")
# - extra_sources_c : List of extra C/C++ source code used for testing (if none, input empty string "")
add_pfunit_test(diagnostics_test ${DIAGNOSTICS_TEST_SRCS} "" "")
# Link the test to the aether library target.
target_link_libraries (diagnostics_test aether)
target_include_directories(diagnostics_test PRIVATE $<TARGET_PROPERTY:aether,Fortran_MODULE_DIRECTORY>)
call set_diagnostics_on(.true.)
call get_diagnostics_on(level)
call check(error, .true., level)
if (allocated(error)) return

end subroutine test_set_and_get

end module test_diagnostics

With our test written we now need to add this into the CMake build generation system.


Add test to CMake
-----------------

In the file *tests/CMakeLists.txt* we need to add the name of our new test, which is *diagnostics* in this case, to the list of tests.
At the top of this file there is a statement to define a variable named *TESTS*, looking something like this::

set(
TESTS
# Other tests may already be present, if so they will appear here separated by white-space.
)

When we add the *diagnostics* test the variable definition should look like this::


set(
TESTS
# Other tests may already be present, if so they will appear here separated by white-space.
"diagnostics"
)

That is all we need to do to add our test into the testing framework.

With our test added to the test framework we can now build and run our test.

Build and run test
------------------

The test we have just completed will be built when we build the configuration from the build directory by default.
That is if we execute the *BUILD_ALL* build target for IDEs like Visual Studio or on *Makefile* generation builds we would simple issue the command *make* in the build directory.
We can also build our test directly be building the target *diagnostics_test*, for *Makefile* generation builds we would issue the command::

make diagnostics_test

To run the test we can execute the ctest command from the command line in the build directory with the following arguments::

ctest -R diagnostics_test
ctest -R diagnostics_test

we will also execute all tests if we execute the command::

ctest
ctest

A handy flag to add to both of these commands is the *--verbose* flag.
This gives us the details output from each test and not just the summary statement.
This gives us the detailed output from each test and not just the summary statement::

ctest -V
2 changes: 1 addition & 1 deletion src/bindings/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ if(SWIG_FOUND)
add_project_config_parameter(Python_ROOT_DIR "" PATH "Define the root directory of a Python installation.")
set(Python_ROOT_DIR ${AETHER_Python_ROOT_DIR})

find_package(Python ${AETHER_PREFERRED_PYTHON_VERSION} COMPONENTS Interpreter Development NumPy)
find_package(Python ${AETHER_PREFERRED_PYTHON_VERSION} COMPONENTS Interpreter Development.Module NumPy)
if(Python_FOUND)
add_subdirectory(python)
else()
Expand Down
12 changes: 2 additions & 10 deletions src/bindings/python/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ set(ADDITIONAL_GENERATED_FILES)
# SWIG has already been found at this point.
include(${SWIG_USE_FILE})

if(${Python_VERSION_MAJOR} STREQUAL "3")
if("${SWIG_VERSION}" VERSION_LESS "4.1.0")
set(PYTHONLIBS_SWIG_FLAGS -py3 -relativeimport)
endif()

Expand Down Expand Up @@ -86,7 +86,7 @@ foreach(SWIG_INTERFACE ${INTERFACE_SRCS})
# so we add the release libraries here for all platforms. This probably means that
# when we try and link to the debug version of the Python libraires on OSX and GNU/Linux
# we will not succeed. But as this is rarely done it hopefully won't become an issue.
target_link_libraries(${MODULE_TARGET} Python::Python Python::NumPy aether_c)
target_link_libraries(${MODULE_TARGET} Python::Module Python::NumPy aether_c)
set_target_properties(${MODULE_TARGET} PROPERTIES
LIBRARY_OUTPUT_DIRECTORY ${PYTHON_PACKAGE_DIR}
RUNTIME_OUTPUT_DIRECTORY ${PYTHON_PACKAGE_DIR}
Expand Down Expand Up @@ -160,11 +160,3 @@ add_custom_command(TARGET ${SWIG_PYTHON_BINDINGS_TARGETS} POST_BUILD
DEPENDS ${CMAKE_CURRENT_SOURCE_DIR}/__init__.py ${AETHER_README_FILE} ${SETUP_PRE_GEN_PY_FILE}
COMMENT "Finalise preparation of Aether Python bindings."
)

if (NOT IS_MULTI_CONFIG AND AETHER_BUILD_TYPE)
string(TOLOWER ${AETHER_BUILD_TYPE} LOWER_AETHER_BUILD_TYPE)
set(_BUILD_TYPE_PART ${LOWER_AETHER_BUILD_TYPE}_)
endif()

find_program(VIRTUALENV_EXECUTABLE NAMES ${VIRTUALENV_PREFERRED_NAMES} virtualenv)
message(STATUS "${VIRTUALENV_EXECUTABLE} -p ${Python_EXECUTABLE} venv_${_BUILD_TYPE_PART}py${Python_VERSION_MAJOR}${Python_VERSION_MINOR}")
3 changes: 3 additions & 0 deletions src/lib/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,9 @@ add_library(aether STATIC ${LIB_SRCS})
set_target_properties(aether PROPERTIES
Fortran_MODULE_DIRECTORY "${AETHER_MODULE_DIRECTORY}"
POSITION_INDEPENDENT_CODE TRUE)

target_include_directories(aether PUBLIC "${AETHER_MODULE_DIRECTORY}")

if (MSVC)
target_compile_options(aether PRIVATE /heap-arrays0)
endif ()
Expand Down
65 changes: 61 additions & 4 deletions tests/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,64 @@

add_subdirectory(pFUnit-3.2.9)
# Unit testing
set(
TESTS
"diagnostics"
)
set(
TEST_SRCS
"testdriver.f90"
)

# Include the config file so we can use the test creation function directly.
include("${CMAKE_CURRENT_BINARY_DIR}/pFUnit-3.2.9/pFUnitConfig.cmake")
set(
TEST_SUITE_USE_STATEMENT
)
set(
TEST_SUITE_LIST_STATEMENT
)

add_subdirectory(lib)
foreach(t IN LISTS TESTS)
string(MAKE_C_IDENTIFIER ${t} t)
list(APPEND TEST_SRCS "test_${t}.f90")
set(TEST_SUITE_USE_STATEMENT "${TEST_SUITE_USE_STATEMENT} use test_${t}, only : collect_${t}\n")
set(TEST_SUITE_LIST_STATEMENT "${TEST_SUITE_LIST_STATEMENT} new_testsuite(\"${t}\", collect_${t}), &\n")
endforeach()

string(REGEX REPLACE ", &\n$" " &" TEST_SUITE_LIST_STATEMENT "${TEST_SUITE_LIST_STATEMENT}")
set(TEST_MAIN "${CMAKE_CURRENT_BINARY_DIR}/testmain.f90")
configure_file(testmain.in.f90 ${TEST_MAIN})
list(APPEND TEST_SRCS ${TEST_MAIN})

add_executable(
"${PROJECT_NAME}-tester"
"${TEST_SRCS}"
)

target_compile_definitions(
"${PROJECT_NAME}-tester"
PRIVATE
"WITH_QP=$<BOOL:${WITH_QP}>"
"WITH_XDP=$<BOOL:${WITH_XDP}>"
)

if (${CMAKE_Fortran_COMPILER_ID} STREQUAL "GNU")
set_source_files_properties(testdriver.f90 PROPERTIES COMPILE_FLAGS -cpp)
endif ()

if (MSVC)
target_compile_options("${PROJECT_NAME}-tester" PUBLIC -fpp)
endif()

target_link_libraries(
"${PROJECT_NAME}-tester"
PRIVATE
aether
)

foreach(t IN LISTS TESTS)
set(_TEST_NAME "${PROJECT_NAME}/${t}_test")
add_test("${_TEST_NAME}" "${PROJECT_NAME}-tester" "${t}")
if(MSVC)
set(_TEST_PROPERTIES "PATH=$<TARGET_FILE_DIR:aether>\;${FORTRAN_RUNTIME_PATH}")
set_tests_properties("${_TEST_NAME}" PROPERTIES ENVIRONMENT "${_TEST_PROPERTIES}" RANDOM_THING $<TARGET_FILE_DIR:aether_c>)
endif()
endforeach()
16 changes: 0 additions & 16 deletions tests/lib/CMakeLists.txt
View file
Open in desktop

This file was deleted.

16 changes: 0 additions & 16 deletions tests/lib/test_diagnostics.pf
View file
Open in desktop

This file was deleted.

Loading