n2n/scripts/test_harness.sh

41 lines
838 B
Bash
Raw Normal View History

added test framework and code coverage reporting (#797) * Add a simple test framework * Add a code coverage report example oneliner * Move the coverage report into a separate directory * Add a github action to run tests and publish a branch with the coverage report * Fix: Missing job separator * Fix: remember to actually run configure * Fix: Gotta autogen before I configure * Dont try to upload coverage report unless this is a push * Clearly show the git ref tested in the coverage report * Add a test for the various transforms * Add tests for the elliptic curve and pearson hash * Ensure we ignore new generated output * Remove unneeded boilerplate from the compression tests * Add an example of a test of the encoded wire packets * Ensure that correctly testable data is output even when zstd is not compiled * Factor test runner out into its own script and attempt to add it to the cmake file * Tell cmake about a new object file * Stop trying to make Cmake work... * Stop trying to make cmake work, round 2 * In the middle of a thousand lines of cmake output was one important one - windows could not find assert() - try again * Try again to plumb the tests into cmake * Add missing library to our superset install line * Fix build error when libcap-dev is installed * Switch to using artifact uploads instead of pages to store/show the coverage report * Fix artifact upload yaml * Upload coverage report to codecov * Fix codecov - clearly it doesnt do a recursive search for coverage files * Fix codecov - my hopeful use of a list of directories didnt work * Fix codecov - unfortunately, it doesnt just consume the coverage data and needs us to generate the gcov output * Fix codecov - nope, it still doesnt recursively search * Fix codecov - it really helps if I run the gcov data generator * Add a simple matrix build * Fix older ubuntu versions of gcovr that do not support the '--html-title' option * Ensure we use gcover options that are identical on older ubuntu * Improve coverage generation and required build packages
2021-09-27 11:41:06 +02:00
#!/bin/sh
#
# This expects to find the tests in the tools dir and the expected results
# in the tests dir.
TESTS="
tests-auth
added test framework and code coverage reporting (#797) * Add a simple test framework * Add a code coverage report example oneliner * Move the coverage report into a separate directory * Add a github action to run tests and publish a branch with the coverage report * Fix: Missing job separator * Fix: remember to actually run configure * Fix: Gotta autogen before I configure * Dont try to upload coverage report unless this is a push * Clearly show the git ref tested in the coverage report * Add a test for the various transforms * Add tests for the elliptic curve and pearson hash * Ensure we ignore new generated output * Remove unneeded boilerplate from the compression tests * Add an example of a test of the encoded wire packets * Ensure that correctly testable data is output even when zstd is not compiled * Factor test runner out into its own script and attempt to add it to the cmake file * Tell cmake about a new object file * Stop trying to make Cmake work... * Stop trying to make cmake work, round 2 * In the middle of a thousand lines of cmake output was one important one - windows could not find assert() - try again * Try again to plumb the tests into cmake * Add missing library to our superset install line * Fix build error when libcap-dev is installed * Switch to using artifact uploads instead of pages to store/show the coverage report * Fix artifact upload yaml * Upload coverage report to codecov * Fix codecov - clearly it doesnt do a recursive search for coverage files * Fix codecov - my hopeful use of a list of directories didnt work * Fix codecov - unfortunately, it doesnt just consume the coverage data and needs us to generate the gcov output * Fix codecov - nope, it still doesnt recursively search * Fix codecov - it really helps if I run the gcov data generator * Add a simple matrix build * Fix older ubuntu versions of gcovr that do not support the '--html-title' option * Ensure we use gcover options that are identical on older ubuntu * Improve coverage generation and required build packages
2021-09-27 11:41:06 +02:00
tests-compress
tests-elliptic
tests-hashing
tests-transform
tests-wire
"
TOOLSDIR=tools
TESTDATA=tests
# Allow both dirs be overidden
[ -n "$1" ] && TOOLSDIR="$1"
[ -n "$2" ] && TESTDATA="$2"
# Confirm we have all the tools and data
for i in $TESTS; do
Basic C Code lint checker and shell checker (#859) * Factor build packages out into a more maintainable list * Create a location for scripts to live * Provide a make target to return the source dir as close as reasonable to the original distributed state * Add a code lint step, checking the coding style * Change test harness as recommended by shellcheck * Ensure we actually have the linter tool installed * Use the correct directory for cmake to run the tests * Adjust for the older uncrustify in the current github ubuntu-latest * Make one file pass the linter * Integrate the lint with the existing test workflow * Add files with minimal changes needed to the linter * Add more files with minimal changes needed to the linter * Dont build binaries if we fail the lint test * Update the phony targets with the lint steps * Ensure the flake8 package is installed in the new lint workflow job * Use the makefile to drive the packages needed to install for linting * No need to add dependancies on lint, just rely on the workflow status to show failure * Update the scripts dir README to reflect current assumptions * Rename and briefly document the indent.sh script * Fix the ignore to ignore the right Makefile * Rename the test_harness script to make it clear it is a shell script * Provide a master lint make target and add a shell script lint tool * Elminate stray tabs * Drop include/auth.h from linter - there are inconsistant results with function definitions when using the current uncrustify rules
2021-10-23 21:36:18 +02:00
if [ ! -e "$TOOLSDIR/$i" ]; then
added test framework and code coverage reporting (#797) * Add a simple test framework * Add a code coverage report example oneliner * Move the coverage report into a separate directory * Add a github action to run tests and publish a branch with the coverage report * Fix: Missing job separator * Fix: remember to actually run configure * Fix: Gotta autogen before I configure * Dont try to upload coverage report unless this is a push * Clearly show the git ref tested in the coverage report * Add a test for the various transforms * Add tests for the elliptic curve and pearson hash * Ensure we ignore new generated output * Remove unneeded boilerplate from the compression tests * Add an example of a test of the encoded wire packets * Ensure that correctly testable data is output even when zstd is not compiled * Factor test runner out into its own script and attempt to add it to the cmake file * Tell cmake about a new object file * Stop trying to make Cmake work... * Stop trying to make cmake work, round 2 * In the middle of a thousand lines of cmake output was one important one - windows could not find assert() - try again * Try again to plumb the tests into cmake * Add missing library to our superset install line * Fix build error when libcap-dev is installed * Switch to using artifact uploads instead of pages to store/show the coverage report * Fix artifact upload yaml * Upload coverage report to codecov * Fix codecov - clearly it doesnt do a recursive search for coverage files * Fix codecov - my hopeful use of a list of directories didnt work * Fix codecov - unfortunately, it doesnt just consume the coverage data and needs us to generate the gcov output * Fix codecov - nope, it still doesnt recursively search * Fix codecov - it really helps if I run the gcov data generator * Add a simple matrix build * Fix older ubuntu versions of gcovr that do not support the '--html-title' option * Ensure we use gcover options that are identical on older ubuntu * Improve coverage generation and required build packages
2021-09-27 11:41:06 +02:00
echo "Could not find test $TOOLSDIR/$i"
exit 1
fi
Basic C Code lint checker and shell checker (#859) * Factor build packages out into a more maintainable list * Create a location for scripts to live * Provide a make target to return the source dir as close as reasonable to the original distributed state * Add a code lint step, checking the coding style * Change test harness as recommended by shellcheck * Ensure we actually have the linter tool installed * Use the correct directory for cmake to run the tests * Adjust for the older uncrustify in the current github ubuntu-latest * Make one file pass the linter * Integrate the lint with the existing test workflow * Add files with minimal changes needed to the linter * Add more files with minimal changes needed to the linter * Dont build binaries if we fail the lint test * Update the phony targets with the lint steps * Ensure the flake8 package is installed in the new lint workflow job * Use the makefile to drive the packages needed to install for linting * No need to add dependancies on lint, just rely on the workflow status to show failure * Update the scripts dir README to reflect current assumptions * Rename and briefly document the indent.sh script * Fix the ignore to ignore the right Makefile * Rename the test_harness script to make it clear it is a shell script * Provide a master lint make target and add a shell script lint tool * Elminate stray tabs * Drop include/auth.h from linter - there are inconsistant results with function definitions when using the current uncrustify rules
2021-10-23 21:36:18 +02:00
if [ ! -e "$TESTDATA/$i.expected" ]; then
added test framework and code coverage reporting (#797) * Add a simple test framework * Add a code coverage report example oneliner * Move the coverage report into a separate directory * Add a github action to run tests and publish a branch with the coverage report * Fix: Missing job separator * Fix: remember to actually run configure * Fix: Gotta autogen before I configure * Dont try to upload coverage report unless this is a push * Clearly show the git ref tested in the coverage report * Add a test for the various transforms * Add tests for the elliptic curve and pearson hash * Ensure we ignore new generated output * Remove unneeded boilerplate from the compression tests * Add an example of a test of the encoded wire packets * Ensure that correctly testable data is output even when zstd is not compiled * Factor test runner out into its own script and attempt to add it to the cmake file * Tell cmake about a new object file * Stop trying to make Cmake work... * Stop trying to make cmake work, round 2 * In the middle of a thousand lines of cmake output was one important one - windows could not find assert() - try again * Try again to plumb the tests into cmake * Add missing library to our superset install line * Fix build error when libcap-dev is installed * Switch to using artifact uploads instead of pages to store/show the coverage report * Fix artifact upload yaml * Upload coverage report to codecov * Fix codecov - clearly it doesnt do a recursive search for coverage files * Fix codecov - my hopeful use of a list of directories didnt work * Fix codecov - unfortunately, it doesnt just consume the coverage data and needs us to generate the gcov output * Fix codecov - nope, it still doesnt recursively search * Fix codecov - it really helps if I run the gcov data generator * Add a simple matrix build * Fix older ubuntu versions of gcovr that do not support the '--html-title' option * Ensure we use gcover options that are identical on older ubuntu * Improve coverage generation and required build packages
2021-09-27 11:41:06 +02:00
echo "Could not find testdata $TESTDATA/$i.expected"
exit 1
fi
done
# Actually run the tests
set -e
for i in $TESTS; do
echo "$TOOLSDIR/$i >$TESTDATA/$i.out"
Basic C Code lint checker and shell checker (#859) * Factor build packages out into a more maintainable list * Create a location for scripts to live * Provide a make target to return the source dir as close as reasonable to the original distributed state * Add a code lint step, checking the coding style * Change test harness as recommended by shellcheck * Ensure we actually have the linter tool installed * Use the correct directory for cmake to run the tests * Adjust for the older uncrustify in the current github ubuntu-latest * Make one file pass the linter * Integrate the lint with the existing test workflow * Add files with minimal changes needed to the linter * Add more files with minimal changes needed to the linter * Dont build binaries if we fail the lint test * Update the phony targets with the lint steps * Ensure the flake8 package is installed in the new lint workflow job * Use the makefile to drive the packages needed to install for linting * No need to add dependancies on lint, just rely on the workflow status to show failure * Update the scripts dir README to reflect current assumptions * Rename and briefly document the indent.sh script * Fix the ignore to ignore the right Makefile * Rename the test_harness script to make it clear it is a shell script * Provide a master lint make target and add a shell script lint tool * Elminate stray tabs * Drop include/auth.h from linter - there are inconsistant results with function definitions when using the current uncrustify rules
2021-10-23 21:36:18 +02:00
"$TOOLSDIR/$i" >"$TESTDATA/$i.out"
cmp "$TESTDATA/$i.expected" "$TESTDATA/$i.out"
added test framework and code coverage reporting (#797) * Add a simple test framework * Add a code coverage report example oneliner * Move the coverage report into a separate directory * Add a github action to run tests and publish a branch with the coverage report * Fix: Missing job separator * Fix: remember to actually run configure * Fix: Gotta autogen before I configure * Dont try to upload coverage report unless this is a push * Clearly show the git ref tested in the coverage report * Add a test for the various transforms * Add tests for the elliptic curve and pearson hash * Ensure we ignore new generated output * Remove unneeded boilerplate from the compression tests * Add an example of a test of the encoded wire packets * Ensure that correctly testable data is output even when zstd is not compiled * Factor test runner out into its own script and attempt to add it to the cmake file * Tell cmake about a new object file * Stop trying to make Cmake work... * Stop trying to make cmake work, round 2 * In the middle of a thousand lines of cmake output was one important one - windows could not find assert() - try again * Try again to plumb the tests into cmake * Add missing library to our superset install line * Fix build error when libcap-dev is installed * Switch to using artifact uploads instead of pages to store/show the coverage report * Fix artifact upload yaml * Upload coverage report to codecov * Fix codecov - clearly it doesnt do a recursive search for coverage files * Fix codecov - my hopeful use of a list of directories didnt work * Fix codecov - unfortunately, it doesnt just consume the coverage data and needs us to generate the gcov output * Fix codecov - nope, it still doesnt recursively search * Fix codecov - it really helps if I run the gcov data generator * Add a simple matrix build * Fix older ubuntu versions of gcovr that do not support the '--html-title' option * Ensure we use gcover options that are identical on older ubuntu * Improve coverage generation and required build packages
2021-09-27 11:41:06 +02:00
done