Fix few limitation which do not allow to run Twister on Windows OS.
Especially turn off "import pty" module, due to the fact, that this
module can be used only on POSIX OS.
Fixes#40007
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Introduce a weak implementation of test_main() which calls:
* ztest_run_registered_test_suites(NULL);
* ztest_verify_all_registered_test_suites_ran();
This will attempt to run all registered test suites and verify that
they each ran.
Signed-off-by: Yuval Peress <peress@chromium.org>
Add new functionality to ztest to improve test modularity. The two
primary new entry points are:
* ztest_register_test_suite
* ztest_run_registered_test_suites
When registering a new test suite, users provide the name as well as
an optional predicate used to filter the tests for each run. Using NULL
as the predicate ensures that the test is run exactly once (after which
it is automatically filtered from future runs).
Calls to ztest_run_registered_test_suites take a state pointer as an
argument. This allows the the pragma functions to decide whether the
test should be run.
The biggest benefit of this system (other than the ability to filter
tests and maintain a larger test state) is the ability to better
modularize the test source code. Instead of all the various tests
having to coordinate and the main function having to know which tests
to run, each source file manages registering its own test
suite and handling the conditions for running the suite.
Signed-off-by: Yuval Peress <peress@chromium.org>
Fixes a bug causing tests with error status get silently skipped
when --subset is used.
Fixes: #39619
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Fixes#38046
Document the changes in twister.rst
Add baud rate to debug message in twister
Add baud parameter to twister's add_device function
Set the twister baud rate from input parameters
Use 115200 as the default baud rate if not specified
Add baud to the hwmap-schema.yaml file
Add --device-serial-baud to twister arguments
Fix compliance issues
Fix mistake in parameter name from device-baud to device-serial-baud
Refactoring of the code in orded to simplify the logic
and clear multiple definitions of the default baud rate.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Signed-off-by: Dennis Ruffer <daruffer@gmail.com>
If timeout error was occured during performing testsuite on QEMU or
other simulator (e.g. Renode), information about this error will be not
placed in final report. It can be fixed, by set status of testcases,
which were not performed due to this error as "BLOCK".
Simmilar solution is alredy used in DeviceHandler when Twister works
with real platform. So I think that it is resonable to use the same
approach, when tests are performed on simulators.
I move loop which iterate through all testcases and set their tests
status as BLOCK into the parent Handler class, because the same
activity is performed in each children class (BinaryHandler,
DeviceHandler and QEMUHandler).
Fixes#38756
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Fixes: #38924
When the `verify-toolchain.cmake` script fails, then twister will print
a standard message to the user regardless of the reason.
> E: Variable ZEPHYR_TOOLCHAIN_VARIANT is not defined
The `verify-toolchain.cmake` already prints detailed information
regarding the cause of the failure, so twister should just pass that
message as-is.
For example, the message that is provided by verify-toolchain.cmake
when Zephyr SDK 0.13.0 is installed but 0.13.1 is required looks like:
> CMake Error at cmake/verify-toolchain.cmake:75 (find_package):
> Could not find a configuration file for package "Zephyr-sdk" that is
> compatible with requested version "0.13.1".
>
> The following configuration files were considered but not accepted:
>
> /opt/zephyr-sdk-0.13.0/cmake/Zephyr-sdkConfig.cmake, version: 0.13.0
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
Tests are reported as skipped if they are only being built. Fix this by
checking for the correct status.
Fixes#37475
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
In the case of native_posix (and other "host" toolchain based
platforms) Twister shouldn't filter tests by ZEPHYR_TOOLCHAIN_VARIANT.
The PR replaces `- zephyr` with `- host` in native_posix(_64).yaml
and modify the filter in twisterlib.py to not block building
if "host" is under the platform supported toolchains.
Fixes: #38418
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
For twister, when simulation is mdb-nsim, the platform is
nsim_hs_smp. Before the twister will call west falsh
when platform is nsim_hs_smp, because twister can't kill
cld process, now this problem has been fixed
Signed-off-by: Jingru Wang <jingru@synopsys.com>
When run by a makefile twister might fail to decode the text results of
commands, based upon the environment settings of the system. With this
change escape characters are removed before the string is passed to the
json decoder.
Signed-off-by: Benedikt Schmidt <benedikt.schmidt@embedded-solutions.at>
When using both test-only and retry-failed flags, all tests would be
added to the test queue regardless if they failed or not. This commit
updates the logic to process test-only runs similar as other runs,
by adding tests to the queue with the 'run' operation directly.
Signed-off-by: Andreas Vibeto <andreas.vibeto@nordicsemi.no>
Create a "global" gen_defines.py option and edtlib.EDT constructor
kwarg that turns edtlib-specific warnings into errors. This applies to
edtlib-specific warnings only. Warnings that are just dupes of dtc
warnings are not affected.
Use it from twister to increase DT testing coverage in upstream zephyr.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Make correction on the terms "intput" appearing
in sciripts/pylib/twister/twisterlib.py to "input" for all.
This type of misspellings may cause another unnoticeable misspellings,
when propagated by code completions.
(assuming that the "intput" was intended
to have the opposite meaning of "output",
which also appear on the same areas)
Signed-off-by: Kentaro Sugimoto <tarotene@gmail.com>
This function takes a node label (not a label property) and returns
true if a node with such label exists and the node is enabled.
Signed-off-by: Andrzej Głąbek <andrzej.glabek@nordicsemi.no>
When integration mode is on, any skips on integration_platforms are
treated as errors. This patch adds an exemption for quarantined tests.
They will stay as skipped.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
If "Cmake build failure" is detected test instance get status "error".
Despite this in final report this error is counted as failure. It can
be fix, by set proper results of each testcase in instance from None
to "BLOCK" after found Cmake error. After this fix, error is counted
properly in final report.
Fixes#37140
Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
In dt_label_with_parent_compat_enabled, it maybe that there is
no node matching expected label.
In this case don't generate error, but return False.
Signed-off-by: Erwan Gouriou <erwan.gouriou@linaro.org>
sometimes ninja fails to launch qemu, the sub-thread
can't read anything from qemu pipeline, then the
corresponding testcase will timeout. Then the
sub-thread will get blocked if it call join()
set terminate as the Handler's method, then
Handler's children class can call it
Signed-off-by: Jingru Wang <jingru@synopsys.com>
Function used for filtering "dt_compat_enabled_with_label" was not
working as expected as it was not taking into account that we're
looking for a children/parent combination:
Provided "compat" with enabled status should be the one of the parent
of the node matching given label.
Function is then reworked to take this into account.
And to make it's usage clear:
- function name is changed to be clearer on the intention
- args order is reversed to be more logical wrt the intention
Users of the function are also updated to take the change into
account.
Fixes#36093
Signed-off-by: Erwan Gouriou <erwan.gouriou@linaro.org>
when generating a new map, set the status of the device and whether it
is connected or now. The available key is runtime only and is not part
of the map anymore.
Adapt documentation and remove available as a key in the generated map.
Fixes#35341
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
On ARC some platforms utilize ICCM/DCCM for their "flash" and "ram"
storage. So we might get errors about overflowing one of these regions.
So treat them similar to how we treat FLASH and SRAM.
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
1. Add a class of Pytest(Harness) which can handle pytest script
in harness.py
2. Use running_dir to store current test directory which be
used in pytest scrpt.
3. Add usage of this feature into zephyr doc.
Signed-off-by: YouhuaX Zhu <youhuax.zhu@intel.com>
All skips for platforms listed in the .yaml's integration_platforms
will be treated as errors in the integration mode ('-G/--integration').
This feature serves to guarantee a proper testing scope in CI as no
integration platform will be skipped silently.
Fixes#33874
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
With the previous logic memory footprint can only be saved in a report
if a given test/sample was fully executed and passed (built and run),
hence build-only tests were not providing these metrics. This commit
modifies the logic so that it is enough to have the build successful
to be able to get the memory footprint.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Upon discovery of a sample/testcase.yaml file, twister will disregard
further subfolders. Given the folder structure below, the testcase.yaml
will currently not be discovered:
samples/sample1/
samples/sample1/sample.yaml
samples/sample1/tests/
samples/sample1/tests/testcase.yaml
This is desirable to allow placing test code closer to sample code.
Signed-off-by: Jorgen Kvalvaag <jorgen.kvalvaag@nordicsemi.no>
Add arm fvp emulator in order to use ninja run or west build -t run.
Add armfvp in order to run twister.
Set env ARMFVP_BIN_PATH before using it,
e.g. export ARMFVP_BIN_PATH=<path/to/fvp/dir>
NOTE: ARMFVP_BIN_PATH is the dir path.
Signed-off-by: Jaxson Han <jaxson.han@arm.com>
Adds feature allowing to use yaml file with dictionaries defining
tests to be quarantined (extra arg "--quarantine-list FILENAME").
The dictionaries are validated according to the proper schema
and loaded.
A flat list is created containing quarantined configurations
(configuration = platform + scenario). Configurations under quarantine
are skipped and get "Quarantine" as a reason in the results reports.
A "comment" can be added to a quarantine entry in the quarantine yaml
with more details (e.g. issue #) and it will be also added to
the report.
The status of tests under quarantine can be verify if
`--quarantine-verify` is used in addition to
"--quarantine-list FILENAME". Using these args will make twister skip
all tests which are not on the quarantine list.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
In CI we typically build for "default" platforms. However some
testcases that set "platform_allow" may have no overlap with the default
platforms and thus thoses testcases will not get built.
Change the platform selection logic in these cases to set the platforms
to the list of allowed platforms ("platform_allow") in the testcase.
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
We are now in the process of extracting edtlib and dtlib into a
standalone source code library that we intend to share with other
projects.
Links related to the work making this standalone:
https://pypi.org/project/devicetree/https://python-devicetree.readthedocs.io/en/latest/https://github.com/zephyrproject-rtos/python-devicetree
This standalone repo includes the same features as what we have in
Zephyr, but in its own 'devicetree' python package with PyPI
integration, etc.
To avoid making this a hard fork, move the code that's being made
standalone around in Zephyr into a new scripts/dts/python-devicetree
subdirectory, and handle the package and sys.path changes in the
various places in the tree that use it.
From now on, it will be possible to update the standalone repository
by just recursively copying scripts/dts/python-devicetree's contents
into it and committing the results.
This is an interim step; do NOT 'pip install devicetree' yet.
The code in the zephyr repository is still the canonical location.
(In the long term, people will get the devicetree package from PyPI
just like they do the 'yaml' package today, but that won't happen for
the foreseeable future.)
This commit is purely intended to avoid a hard fork for the standalone
code, and no functional changes besides the package structure and
location of the code itself are expected.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Remove dead code from twisterlib.py which is a copy of code
from twister script. This redundancy causes pylint errors in
compliance checks.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
The function `handle()` from class `Test(Harness)` had most of its
code being the direct copy of the code from
`Harness.process_test(line)`. This patch replaces the copied code
with the call for `process_test(line)` and removes repetition of
this function call.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Use edt.label2node in dt_compat_enabled_with_label DT filter.
Signed-off-by: Johann Fischer <johann.fischer@nordicsemi.no>
Co-authored-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
The patch improves handling runners errors from flashing. It
add thread termination in case of process returning non 0 exit
code. Before, even though the error was returned,
a thread was not terminated and test ended
only after timeout termination. The patch also adds information about
the failure reason to the reports
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
The function returns true if the node with the referenced
node label is compatible and enabled.
Signed-off-by: Johann Fischer <johann.fischer@nordicsemi.no>
Add execution time for testing result of each ztest testcase as:
START - test_sem_multi_take_timeout_diff_sem
PASS - test_sem_multi_take_timeout_diff_sem in 2.54 seconds
Fix#32137.
Signed-off-by: Enjia Mai <enjiax.mai@intel.com>
Logic for selecting non-default platforms listed in
integration_platforms did not work. Functionality is restored in this
change.
Fixes#32835
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Originally added in 7733b94224.
This filter is not well-formed. It's meant to match nodes like
/leds/led_0 in this DTS:
/ {
aliases {
led0 = &led0;
};
leds {
compatible = "gpio-leds";
led0: led_0 {
gpios = <...>;
label = "LED 0";
};
};
};
Uses look like this:
filter: dt_compat_enabled_with_alias("gpio-leds", "led0")
But notice how the led_0 node doesn't have compatible "gpio-leds";
it's actually the *parent* node that has that compatible.
Replace this with a new filter, dt_enabled_alias_with_parent_compat(),
which is used like this:
filter: dt_enabled_alias_with_parent_compat("led0", "gpio-leds")
This has a name and argument order that makes the meaning of the
filter clearer.
Replace in-tree users with the new filter.
Deprecate the old filter and warn about its use using the standard
logging module.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
CPython is sometimes described as "single threaded" due to the GIL,
but the interpreter will still "preemptively" switch between threads
(the details seem poorly documented).
So the time between checking whether acquire is 1 and decrementing the
count could result in more than one thread seeing an "available"
device, and more than one test being run (simultaneously, on the same
physical device!). We have a big herd of threads all polling for
this, so in a large test run this would happen maybe one time out of
20-30 tries.
Use a lock. Also remove the very similar looking
DUT.get_available_device() method, which had the same bug but appears
to be dead code.
Fixes#32679
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
Use EXTRA_GEN_DEFINES_ARGS to error out on deprecated devicetree
properties when warnings are treated as errors.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
sometimes a test instance hasn't been boot up to execute successfully,
so the results is an empty dictionary, it will not be added into final
twister_report.xml file.
in order to include it, fill the results as BLOCK.
Signed-off-by: peng1 chen <peng1.chen@intel.com>
Currently duplicated testcases are only determined by the classname.
This results in testcases not beeing added to the twister.xml if the
testcases are from the same class (e.g. sample.yaml), but with different
names.
The changed check if the combination of classname and testname
already exists.
Signed-off-by: Sebastian Wezel <sebastian.wezel@nordicsemi.no>
Include coverage support for device testing. It works by switching
capture_coverage to True (as it would meet coverage data start
pattern). Then we continue tot read coverage data until we get
coverage data end pattern. Otherwise, after receiving test result
pattern, we close serial console and do not get time to find start
coverage data pattern.
Signed-off-by: Andrei Emeltchenko <andrei.emeltchenko@intel.com>
Fix twister output. "Test suite" used previously did not correspond
to what was counted (test scenarios). A test scenario is an entry from
testcase/sample.yaml under "tests:", which specify configuration used to
build a test. Changes in docs will follow later on. Also replace the
amount of preselectd platforms with the amount of platforms that
were not filtered out and actually tested.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Followup commit on comment in #32003.
Using pathlib to avoid mixing of `/` and `\` if twister someday will be
able to run on Windows.
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
Piotr Golyzniak