Currently in binary handler and qemu handler we have status `failed`
in case of test timeout, but in device handler we have status `error`
This not only adds inconsistency between handlers, but also prevents
us from usage test retry functionality for the runs on HW.
Fix timeout status by changing it to `failed` instead of `error`.
Signed-off-by: Eugeniy Paltsev <Eugeniy.Paltsev@synopsys.com>
Signed-off-by: Evgeniy Paltsev <PaltsevEvgeniy@gmail.com>
In many of test specifications yaml files (testcase.yaml, sample.yaml)
section `platform_allow` is added. This change allows to test some
scenarios on platforms, that are not added yet to platform allow
list (or are not going to be added for some reasons).
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Fix how DeviceHandler class does 'serial leftover' cleanup
with loss of test output observed on fast console connections,
e.g. over telnet: either 1 sec. of output might be missed,
or up to 1000 bytes/chars, whatever happens first.
Signed-off-by: Dmitrii Golovanov <dmitrii.golovanov@intel.com>
We are wrongly claiming no console output, yet console output is there
and the failure is just a regular timeout, i.e. the test did not
complete within allocated time.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Add support test levels and the ability to assign a specific test to one
or more levels. Using command line options of twister it is then possible
to select a level and just execute the tests included in this level.
Additionally, a test configuration allows definiing level
dependencies and additional inclusion of tests into a specific level if
the test itself does not have this information already.
In the configuration file you can include complete components using
regular expressions and you can specify which test level to import from
the same file, making management of levels easier.
To help with testing outside of upstream CI infrastructure, additional
options are available in the configuration file, which can be hosted
locally. As of now, those options are available:
- Ability to ignore default platforms as defined in board definitions
(Those are mostly emulation platforms used to run tests in upstream
CI)
- Option to specify your own list of default platforms overriding what
upstream defines.
- Ability to override build_onl_all options used in some testscases.
This will treat tests or sample as any other just build for default
platforms you specify in the configuation file or on the command line.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Twister allows filtering based on kconfigs and dts, however the
filtration is a part of the cmake stage, i.e. the stage has to pass
first and then twister checks if required properties are available.
This causes problems, when the full cmake stage is unable to pass.
If so, other filtration methods had to be used, e.g. platform_allow.
The commit modifies the twister workflow:
if a test defines filters based on kconfig/dts first a cmake package
helper script is used to extract dt and/or kconfigs and if the
conditions are fulfilled it proceeds to a regular cmake stage.
If not, test is skipped.
Co-authored-by: Daniel DeGrasse <daniel.degrasse@nxp.com>
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
This is an implementation of issue #48334 and adds support for
specifying additional config and device tree overlays through fields in
the testcase.yaml file, which is more readable than having to cram these
in to `extra_args`.
Consider this example which shows the original and new ways to add
config and DT overlays:
```
common:
extra_args: "CONF_FILE=a.conf;b.conf
DTC_OVERLAY_FILE=w.overlay;x.overlay OVERLAY_CONFIG=e.conf
UNRELATED=abc"
tests:
utilities.base64:
extra_conf_files:
- "c.conf"
- "d.conf"
extra_overlay_confs:
- "extra_overlay.conf"
extra_dtc_overlay_files:
- "y.overlay"
- "z.overlay"
extra_configs:
- CONFIG_SAMPLE=y
tags: base64
type: unit
```
The new fields are `extra_conf_files`, `extra_overlay_confs,
`extra_dtc_overlay_files`. Files specified in these sections are appended
to any `CONF_FILE`, `OVERLAY_CONFIG`, or `DTC_OVERLAY_FILE` fields in
`extra_args`, causing the following args being passed in to
`self.run_cmake` at `runner.py:850`:
```
['-DUNRELATED=abc',
'-DCONF_FILE=a.conf;b.conf;c.conf;d.conf',
'-DDTC_OVERLAY_FILE=w.overlay;x.overlay;y.overlay;z.overlay',
'-DOVERLAY_CONFIG=e.conf extra_overlay.conf '
'<build_dir>/twister/testsuite_extra.conf']
```
These fields can be used in the common or scenario-specific YAML
sections and will be merged in order of least to most specific:
1. config files extracted from common's extra_args
2. files listed in common's {extra_conf_files or extra_overlay_confs
or extra_dtc_overlay_files}
3. config files extracted from test scenario's extra_args
4. files listed in test scenario's {extra_conf_files or
extra_overlay_confs or extra_dtc_overlay_files}
Specifying these files in extra_args now triggers a deprecation warning,
as the direct YAML fields are preferred for readability. They will still
function for now but support will be dropped in the future. One
testcase.yaml
(`zephyr/tests/cmake/overlays/var_expansions/testcase.yaml`) is
converted to use the new fields. A follow-up PR will convert the
remaining files to the new format.
Signed-off-by: Tristan Honscheid <honscheid@google.com>
Percentage calculation was off due to a recent fix in counting and lines
were not breaking correctly on errors.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Fixed runtime cleanup option. A typo in a conditional prevented the
cleanup from happened when a test has passed.
Fixes#54240
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Initialize 'use_system_gcov', so it is always set before being referenced
in "elif" statement.
Fixes: 1440b19bde ("twister: restore logic to determine which gcov we
should use")
Signed-off-by: Marcin Niestroj <m.niestroj@emb.dev>
Stop counting errors as failures and report them as is. This fixes
issues with counting when retrying failures and errors.
We have been adding errors to failures and then double count them when
displaying information on the screen instead of keep an accurate count
in the counter class.
This is now fixed and we do report errors seperately, ie. build errors.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
We do not want to use gcov from the SDK for native posix and unit tests,
instead we want the system gcov. Restore logic that determines that.
Fixes#54226
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Move extra warning option from generic twister script into
compiler-dependent config files.
ARCMWDT compiler doesn't support extra warning options ex.
"-Wl,--fatal-warnings". To avoid build fails flag
"disable_warnings_as_errors" should be passed to twister.
This allows all warning messages and make atomatic test useles.
Signed-off-by: Nikolay Agishev <agishev@synopsys.com>
Tests under quarantine are not treated as errors in the integration
mode. However --quarantine-verify argument allowing to execute just
test under quarantine and skipping others was not considered in
relation to integration mode. Those skips are wrongly threated as
errors. This commit fix this relation and makes thoses skips not
turned to errors.
fixes: #54516
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Take an instance that has failed and not whatever was last processed in
the for loop.
Also, change instructions to use `west twister` rather than
`./scripts/twister` to be consistent.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
If errors are found and tests are failing, print a brief summary showing
the top 10 items failing with instructions on how to reproduce either
with twister or west.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Disables allowing the python argparse library from automatically
shortening command line arguments, this prevents issues whereby
a new command is added and code that wrongly uses the shortened
command of an existing argument which is the same as the new
command being added will silently change script behaviour.
Signed-off-by: Jamie McCrae <jamie.mccrae@nordicsemi.no>
When twister is started from make, use the information in MAKEFLAGS to
participate in the job server started by make.
When twister is running standalone, start a jobserver to share with
child processes. If using the kitware version of ninja, then this will
prevent ninja+twister from starting more processes than XX when twister
is run with -jXX.
There is no implementation for windows yet.
Signed-off-by: Jeremy Bettis <jbettis@google.com>
This reverts commit 21eb27c5c0.
This change has been a source of much trouble and ends up preventing us
from tracking test results across tree. It uses the repo name as the
namespace, and that is not always the same and does not have to be
called 'zephyr', depending on where you are running and in which
environment.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Improves the message for tests being excluded using the platform key,
describes the key already found and the platform being used to run the test
in its place.
Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
Implementation ported from TwisterV2.
- quarantine handled by separate module
- multiple yaml allowed from args: --quarantine-list
- scenarios, platforms, architectures keywords in
quarantine yaml are optional, if not given - means take it all
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Some hardware boards are supported in qemu as simulation platforms. If
we are testing with the real hardware, make sure we treat this as a
device and not as a simulator. First priority when initializing a
handler is to check if device testing is requested, then we can handle
the rest without --device-testing.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Adds an option to inform twister a testsuite should only be built and
run for platforms with unique sets of attributes. This enables
for example keying on unique (arch, simulation) platforms to run the test
suite on.
The most common usage may be test suites configured to run once per
(arch, simulation) pair as being enough. Additional information about
platforms may enable running a test once per hardware IP block or once
per soc family or soc avoiding duplicated effort in building and running
tests when once suffices.
Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
We want to be able to have platform or architecture extra configs
without having to duplicate a whole section of the test specification.
This adds support for namespacing of extra configs, for example:
arch:nios2:CONFIG_SYS_CLOCK_TICKS_PER_SEC=1000
or
platform:qemu_x86:CONFIG_FOO=y
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Previous commit dealing with cleanup (c304db56f1) was not taking
--prep-artifacts-for-testing into consideration. Fix this and make sure
we do cleanup when this option is enabled.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Have boards define what binaries should be kept for device testing. The
purpose of this is to reduce the amount of artifacts and just keep the
binaries needed for flashing a device, reducing the size of the final
payload.
A board can have twister keep multiple binaries needed for testing when
--prep-artifacts-for-testing option is used.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
When re-generating hardware map for DKs with more than 1 serial port (more
enries in the hardware map), then we keep enfo only about the last entry.
For boards like nrf5340 it works, because the last serial port is used,
but for nrf52840 with two ports, the first one is important, so after
re-generating the hardware map it does not work.
The solution is to keep all entries after re-generating the hardware map.
Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
When scanning for available elfs, twister discards ones with `_pre`
in their name, to not include prebuilds like 'zephyr_pre0.elf`.
However, the scan looks at the whole test name. If _pre is in the name
(e.g. _precision) it would be wrongly discarded ending with twister
reporting error that no elfs were found. This commit makes the discard
to look only at the last part of the name (i.e. not including test name
but only a lowest level, where zephyr.elf etc are located.)
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
When using llvm we need to set the gcov-tool to "llvm-cov gcov" but
the lcov tool is incapable of passing arguments to the gcov-tool. i.e.
the following cannot work:
$ lcov --gcov-tool "llvm-cov gcov" ...
Instead, create a symlink to llvm-cov prefixed as `gcov` which by the
documentation of llvm-cov will alias to `llvm-cov gcov` subcommand.
Signed-off-by: Yuval Peress <peress@google.com>
This commit expands Twister's CLI so that arguments passed after
a double dash (`--`) get forwarded to the ztest test executable when
run. When built for native_posix, ztest provides a useful CLI that
allows filtering individual suites and tests, adjusting timing settings,
and controlling the shuffling seed. Currently there is no easy way to
use these (with the exception of `--seed`) without needing to dig out
the resulting `zephyr.exe` binary from the build dir and invoke it
manually.
### Examples
Run a specific ztest suite only (useful when writing tests and you don't
want to run a long testcase in its entirety)
```
$ scripts/twister \
-p native_posix \
-s
zephyr/tests/lib/cmsis_dsp/complexmath/libraries.cmsis_dsp.complexmath \
-- -test=complexmath_q31::*
```
Unrecognized arguments that precede the double dash will result in an
error message:
```
$ scripts/twister \
-p native_posix \
--foobar 123 \
-- -test=basic_math_q7::*
...
Unrecognized arguments found: '--foobar 123'. Use -- to delineate extra
arguments for test binary or pass -h for help.
```
Signed-off-by: Tristan Honscheid <honscheid@google.com>
Ignore unit tests and EFI binaries when calculating footprints with
--show-footprint, or we will get a build error.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
The --snr (nrfjprog) --id (jlink) and --board-id (pyocd) options were
deprecated a long time ago in favor of --dev-id. It is time to remove
them.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
This commit calculate memory footprint from build.log and
proposes an alternative approach to #2465.
Signed-off-by: Katarzyna Giądła <katarzyna.giadla@nordicsemi.no>
non-simulation platforms set key to 'na', we need to check for that
instead of just checking for existance of key.
Fixes#52595
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
If a default platform is not able to run for whatever reason, treat it
as any other platform. Default platforms shall ALWAYS be runnable in CI,
or else we are just re-building for no good reason.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Evgeniy Paltsev