Add a test to make sure that when a zassume fails, we mark the test
as failed and print an "Assumption failed" error for the test
Signed-off-by: Yuval Peress <peress@google.com>
Adding the new Kconfig (enabled by default) to make a failed assumption
mark the final result as failed. This change has the following benefits
which have been asked for by the Zephyr community:
1. A failed assumption does not go silent. In this example, the failed
assumption will still mark the test as skipped, but the final result
will be to mark the full test run as failed. This would allow
blocking the CI when an assume fails.
2. Normal test skipping via the ztest_test_skip() is unaffected by this
change. Those tests will be marked as skipped, but the binary will
still pass.
Signed-off-by: Yuval Peress <peress@google.com>
I've update FFF to run tests in the native CI for GitHub, so we no
longer need to run these extra tests. Remove the fff module from
west.yml since the module was only used in the CI and the header
was directly included in the Zephyr tree.
See https://github.com/zephyrproject-rtos/fff/pull/2
Signed-off-by: Yuval Peress <peress@google.com>
When CONFIG_ZTEST_SHUFFLE is enabled and ASSERTS are disabled
`start_pos` becomes an unused variable leading to a compile error.
Cleaned-up shuffling algorithm to not need a `start_pos` check.
Signed-off-by: Al Semjonovs <asemjonovs@google.com>
This commit disables the `testing.ztest.ztress` test on
`qemu_cortex_r5` because it fails at run-time when compiled with
GCC 12.
Revert this commit when the GitHub issue #49494, which tracks this bug,
is fixed.
Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
Add specialized tests that execute another test indirectly and ensure
that the results report errors correctly. See the README.rst file
provided with this commit for details.
Signed-off-by: Yuval Peress <peress@google.com>
With the support of Kconfig in unittest CMake module then prj.conf is
now supported.
Add a prj.conf with ZTEST config settings and remove hardcoded values
from CMakeLists.txt file.
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
Add test summary after all test suites finish running.
The summary can be one-line or verbose, which is configured
with CONFIG_ZTEST_VERBOSE_SUMMARY. The one-line summary covers
overall suite stats. The verbose summary covers each test
function within the suite besides the one-line summary.
The new ztest output ultimately go through the printk. If
printk go through the logging subsystem, there may be log
messages dropped. And if log_panic is invoked, log messages
can be flushed in a mess. So several explicit log flush
are used when printing summary to ensure no content is lost
and content is in good shape.
Some macros are shared between old and new ztests. Such as
TC_START_PRINT and TC_END_PRINT. The are defined accordingly.
Signed-off-by: Ming Shao <ming.shao@intel.com>
This commit enables the assert test mode (`CONFIG_ASSERT_TEST`) for the
ztest error hook test because it implements a custom post assert fail
hook (`ztest_post_assert_fail_hook`) that returns without aborting to
faciliate the testing of the assert functions.
Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
Updates the ztest_test_fail() function to allow failures in setup.
When executed, a failed assert will fail every test in the suite owning
the setup function. This was verified by adding a suite which asserts
in the setup function and has a test that should pass. During
exeuction, ztest marks the test as failing.
In order to verify exection I also added 2 new APIs:
- ZTEST_EXPECT_FAIL(suite_name, test_name)
- ZTEST_EXPECT_SKIP(suite_name, test_name)
Signed-off-by: Yuval Peress <peress@google.com>
Add a bunch of missing "zephyr/" prefixes to #include statements in
various test and test framework files.
Signed-off-by: Fabio Baltieri <fabiobaltieri@google.com>
Add an assume API which works like JUnit's. Assumptions can be made
at any point before your test returns (setup, before, and during the
test). If an assumption fails, the test will be marked as skipped.
This helps avoid a cascading affect of failed tests when a base
feature breaks. The feature is expected to have tests and the tests
which depend on it should be skipped (not failed) if that feature
is broken.
Issue #42472
Signed-off-by: Yuval Peress <peress@google.com>
Fix the deprecated calls to loading ZephyrUnittest by replacing them
with `Zephyr COMPONENTS unittest`.
Signed-off-by: Yuval Peress <peress@google.com>
Since output can be cumstomized, so slightly change the output format,
so that twister can parse the output and collect the passrate correctly.
Signed-off-by: Guo Lixin <lixinx.guo@intel.com>
Adds few missing zephyr/ prefixes to leftover #include statements that
either got added recently or were using double quote format.
Signed-off-by: Fabio Baltieri <fabiobaltieri@google.com>
Update the new API to use K_USER as the flags for both
CONFIG_USERSPACE and CONFIG_TEST_USERSPACE. Also, fix the linker
script to properly include the suites, tests, and rules.
Fixes#44108
Signed-off-by: Yuval Peress <peress@google.com>
1.Remove filter to allow platforms(e.g. qemu_x86) supporting
CONFIG_ARCH_HAS_USERSPACE run testcases when userspace is disabled.
2.Define all testcases in the yaml for consistency and issue parsing
them during setup.
Signed-off-by: Guo Lixin <lixinx.guo@intel.com>
Instead of relying on runtime filter to limit scope to emulation
platforms, use the type attribute for each platform and do the filtering
very early on. This will speed things up for tests where we only run on
emulation platforms.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Enable ZTEST_DO_THE_SHUFFLE to shuffle the order tests are ran.
Additional configs ZTEST_DO_THE_SHUFFLE_SUITE_REPEAT_COUNT
ZTEST_DO_THE_SHUFFLE_TEST_REPEAT_COUNT specify the number of times
the test or suite is executed.
Signed-off-by: Al Semjonovs <asemjonovs@google.com>
In order to bring consistency in-tree, migrate all tests to the new
prefix <zephyr/...>. Note that the conversion has been scripted, refer
to #45388 for more details.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
Some names of the test cases are duplicated within the project.
This commit contains the proposed names of the test scenarios.
Signed-off-by: Katarzyna Giadla <katarzyna.giadla@nordicsemi.no>
The filter in the testcase.yaml made it impossible for these to run.
Also, the Kconfig names for the options didn't really make much sense
so they were renamed to avoid conflict.
This fixes an issue found by #36433.
Signed-off-by: Yuval Peress <peress@google.com>
Test were executed on single CPU only and with qemu_cortex_a9
excluded. Removing those limitations after fixes.
Signed-off-by: Krzysztof Chruscinski <krzysztof.chruscinski@nordicsemi.no>
Seen failures on some platforms. No harm to relax the
check for test timeout.
Signed-off-by: Krzysztof Chruscinski <krzysztof.chruscinski@nordicsemi.no>
Change the CPU_CORTEX_R kconfig option to CPU_AARCH32_CORTEX_R to
distinguish the armv7 version from the armv8 version of Cortex-R.
Signed-off-by: Bradley Bolen <bbolen@lexmark.com>
Anas Nashif