Use the pytest test framework in the dtlib.py and edtlib.py test
suites (testdtlib.py and testedtlib.py respectively).
The goal here is not to change what is being tested. The existing test
suite is excellent and very thorough.
However, it is made up of executable scripts where all of the tests
are run using a hand-rolled framework in a single function per file.
This is a bit all-or-nothing and prevents various nice features
available in the de-facto standard pytest test framework from being
used.
In particular, pytest can:
- drop into a debugger (pdb) when there is a problem
- accept a pattern which specifies a subset of tests to run
- print very detailed error messages about the actual and expected
results in various traceback formats from brief to very verbose
- gather coverage data for the python scripts being tested (via plugin)
- run tests in parallel (via plugin)
- It's easy in pytest to run tests with temporary directories
using the tmp_path and other fixtures. This us avoid
temporarily dirtying the working tree as is done now.
Moving to pytest lets us leverage all of these things without any loss
in ease of use (in fact, some things are nicer in pytest):
- Any function that starts with "test_" is automatically picked up and
run. No need for rolling up lists of functions into a test suite.
- Tests are written using ordinary Python 'assert'
statements.
- Pytest magic unpacks the AST of failed asserts to print details on
what went wrong in really nice ways. For example, it will show you
exactly what parts of two strings that are expected to be equal
differ.
For the most part, this is a pretty mechanical conversion:
- extract helpers and test cases into separate functions
- insert temporary paths and adjust tests accordingly to not match
file names exactly
- use 'assert CONDITION' instead of 'if not CONDITION: fail()'
There are a few cases where making this happen required slightly
larger changes than that, but they are limited.
Move the checks from check_compliance.py to a new GitHub workflow,
removing hacks that are no longer needed.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This commit introduces support for multiple SOC_ROOT.
This means that additional SOC_ROOTs specified using -DSOC_ROOT as
argument to CMake will be forming a list together with ${ZEPHYR_BASE}.
This allows for greater flexibility, as developers can now specify
multiple out-of-tree SoCs and not worry about the SoC used for the
board they compile for.
Also it avoid code, such as:
if(BOARD STREQUAL my_board_using_out_of_tree_soc)
set(SOC_ROOT some/out/of/tree/soc/path)
endif()
in application CMakeLists.txt.
Finally, allowing multiple SOC_ROOTs prepares for specifying SOC_ROOTs
in Zephyr modules.
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
Now that kconfigfunctions.py is loading the EDT object from a pickle
file, I can't find any other reads of the environment variables it was
previously using to do that.
(The CMake variables with the same names are read in different places;
I'm specifically talking about environment variables here.)
Remove the dead stores.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
BT simulator tests run standalone and independent of sanitycheck and
might be triggered by tests that are not managed by sanitycheck, so
leave it alone.
Fixes#26508
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
* Tweak west_setup:
- log `west update` to a file to reduce noise in log
- use `west forall` + `git reset` to make sure files are checked out
(this is to handle a possible module cache)
* Output when we start sanity_check. Add a banner for when we run
sanity_check so it's a bit easier to find in console logs
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
When files were added, matching did not work and we were counting added
files twice. Use sets to make sure we have a unique list of files.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
if we are changing a test or a sample, there is not need to run
everything, it is enough to just build/run the changed test/sample.
Also, some files have impact on the code and if they are being changed,
do not run fully sanitycheck.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Script to detect if full sanitycheck should run or if we can skip it and
just run the code that actuallt changed (in samples/tests).
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
When merging files, make sure to remove header rows from each file and
just keep one header. This will stop us from parsing header rows as
tests.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Replace running west command tests in run_ci.sh with a github workflow.
This provides some benefits in that we can run the west command tests on
multiple python versions and host OSes (linux, mac and windows).
Also have the benefit that the tests are only run on modifications to
files in scripts/west_commands/ or scripts/west-commands.yml.
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
So that the EDTT tool could be used by this script,
set its path so other scripts could find it
Signed-off-by: Alberto Escolar Piedras <alpi@oticon.com>
With timestamps enabled, the log in shippable is very difficult to read,
so remove those and enable them when needed for debugging only.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
If modules announce they have tests, run sanitycheck on those modules.
This will run on first matrix node and as the last step after running
regular testsuite.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Fetch HW models from a new west module.
And, remove all pre-west glue which was used to:
* Fetch them in CI
* Validate their vesion
* Modify the include path and link to them
Signed-off-by: Alberto Escolar Piedras <alpi@oticon.com>
Make sure we cleanup the manifest files at the right spot of the CI
process. When not building a PR, behavior is different.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
For cases like zephyr-test or for other reasons the repo dir might not
be called zephyr. Just use the basename of the dir instead of assuming
its called zephyr.
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
pylint does not like how this library works and generates spurious
warnings like
scripts/ci/get_modified_tests.py:55:13: E1121: Too many positional
arguments for function call (too-many-function-args)
scripts/ci/get_modified_tests.py:57:13: E1123: Unexpected keyword
argument '_tty_out' in function call (unexpected-keyword-arg)
These warnings are useful enough to want to have enabled in the upcoming
pylint CI check, so suppress them here.
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
When running run_ci.sh locally with:
./scripts/ci/run_ci.sh -l -b master -R upstream/master..
we export BSIM_OUT_PATH unconditionally which causes sanitycheck to fail
on nrf52_bsim (it depends on BSIM_OUT_PATH variable).
Check if the path defined in BSIM_OUT_PATH is available and unset env.
variable if it is not.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
The main change is the elimination of the bootstrapper, a design flaw
/ misfeature.
Update the documentation to be compatible with the 0.6.x releases as
well. This has to be done atomically, as there were incompatible
changes. Make use of the versionchanged and versionadded directives
to begin keeping track of how these APIs are evolving.
(Note that west 0.6.0 will remain compatible with the extension
commands in Zephyr v1.14 LTS as long as that is still alive. This
change is targeted towards Zephyr 2.0 users.)
This requires a bump in the shippable container and allows us to
simplify the west_commands test procedure.
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
Found a few annoying typos and figured I better run script and
fix anything it can find, here are the results...
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
We not analysing coverage data at all in CI right now. Disable this
while we figure out a better solution for reporting data.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
When detecting changes to boards, make sure we test all board
configurations available in that board directory, not only the main one.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Update the files which contain no license information with the
'Apache-2.0' SPDX license identifier. Many source files in the tree are
missing licensing information, which makes it harder for compliance
tools to determine the correct license.
By default all files without license information are under the default
license of Zephyr, which is Apache version 2.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
The code coverage reports on pull requests has been inconsistent and
confusing. Disable them for pull-requests.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Fixes pylint warnings like this one:
doc/conf.py:325:0: W1401: Anomalous backslash in string: '\s'.
String constant might be missing an r prefix.
(anomalous-backslash-in-string)
The reason for this warning is that backslash escapes are interpreted in
non-raw (non-r-prefixed) strings. For example, '\a' and r'\a' are not
the same string (first one has a single ASCII bell character, second one
has two characters).
It just happens that there's no \s (or \., or \/) escape for example,
and '\s' turns into two characters (as needed for a regex). It's risky
to rely on stuff like that regexes though. Best to make them raw strings
unless they're super trivial.
Also note that '\s' and '\\s' turn into the same string.
Another tip: A literal ' can be put into a string with "blah'blah"
instead of 'blah\'blah'.
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
Martí Bolívar