This PR splits the CI Build Job sim-01 and adds sim-03:
Before the Split: Simulator Jobs take up to 1.5 hours to complete
- sim-01 (1 hour 31 mins): adb, citest, lvgl, matter
- sim-02 (28 mins): posix_test, sqlite
After the Split: Simulator Jobs will complete within 1 hour
- sim-01 (58 mins): adb, citest
- sim-02 (35 mins): lvgl, matter
- sim-03 (28 mins): posix_test, sqlite
This will help us comply with the ASF Policy for GitHub Actions, as explained here: https://github.com/apache/nuttx/issues/14376
This PR creates the new CI Build Jobs `arm64-01` and `x86_64-01`. The new jobs will split and offload the Arm64 and x86_64 Build Targets from `other`. This will reduce our usage of GitHub Runners, to comply with the ASF Policy for GitHub Actions. (Recently we see more PRs for Arm64 and x86_64)
Before the Split: Simple PRs (One Arch and/or One Board) for Arm64 and x86_64 require almost 1 hour for CI Build
- `other` (57 mins): AVR, SPARC, x86, PinePhone, QEMU Arm64, QEMU x86_64
After the Split: Simple PRs for Arm64 and x86_64 will complete under 30 mins
- `other` (24 mins): AVR, SPARC, x86
- `arm64-01` (29 mins): PinePhone, QEMU Arm64
- `x86_64-01` (9 mins): QEMU x86_64
To skip more unnecessary builds: Our Build Rules `arch.yml` shall ignore the label "Area: Documentation", so that a Simple PR + Docs is still a Simple PR. Previously we experienced longer CI Build Times, just because we added docs to our Simple PR. (Now our PR shall be built exactly like a Simple PR)
The updated CI code is explained here: https://github.com/apache/nuttx/issues/13775
To speed up the CI Workflow, this PR splits the CI Build Jobs for RISC-V into smaller jobs. Each job will now complete within 1 hour.
Before the PR: There are 2 jobs for RISC-V, each requiring more than 1.5 hours
- `risc-v-01` (1 hour 42 mins): BL602, Ox64, ESP32-C3 / C6 / H2
- `risc-v-02` (1 hour 41 mins): K230, Icicle, QEMU, RV32M1-Vega
After the PR: The build is spread across 6 jobs for RISC-V, each job completes within 1 hour
- `risc-v-01` (19 mins): BL602, Ox64
- `risc-v-02` (44 mins): ESP32-C3
- `risc-v-03` (45 mins): ESP32-C6, ESP32-H2
- `risc-v-04` (31 mins): K230, Icicle
- `risc-v-05` (41 mins): QEMU CITest
- `risc-v-06` (38 mins): Rest of QEMU, RV32M1-Vega
Following the same convention as the Arm32 Build Jobs, the above jobs are sorted by Target Name. Performance of the RISC-V Build Jobs is discussed in https://github.com/apache/nuttx/issues/13775
CI Build Job `arm-05` (runtime 2 hours) has become the Performance Bottleneck for CI Workflow. That's because `arm-05` builds too many targets for nRF, RP2040, SAM 3, SAM A and SAM D. This PR splits `arm-05` into multiple smaller jobs, to reduce the CI Build Duration.
Before the PR: `arm-05` is overloaded, build requires 2 hours
- `arm-05` (2 hours): nRF, RP2040, SAM 3, SAM A, SAM D
- `arm-06` (56 mins): STM32 [a-m]*
After the PR: `arm-05` is offloaded (to `arm-06` and `arm-07`), completes within 1 hour
- `arm-05` (47 mins): nRF
- `arm-06` (1 hour): Reserve for RP2040 exclusively
- `arm-07` (1 hour 15 mins): SAM 3, SAM A, SAM D, STM32 [a-m]*
Build Jobs are sorted by Target Name. So we cascade the changes and rename the Build Jobs: `arm-07` becomes `arm-08`, `arm-08` becomes `arm-09` etc. Then `arm-13` becomes a new job `arm-14`. (Which we added to `build.yml`)
Performance of `arm-05` is discussed in https://github.com/apache/nuttx/issues/13775 and https://github.com/apache/nuttx/issues/12773
This PR continues to enhance the CI Workflow, to skip the unnecessary NuttX Builds. The PR will update the CI Build Workflow `build.yml`, to call the Refactored Build Rules in `arch.yml` (which is a Reusable Workflow).
The original rules were migrated from `build.yml` to `arch.yml`:
- We target only the Simple PRs: One Arch Label + One Size Label (e.g. "Arch: risc-v, Size: XS")
- For "Arch: risc-v": Build `risc-v-01`, `risc-v-02`
- For "Arch: xtensa": Build `xtensa-01`, `xtensa-02`
- The above rules apply when the PR is Created or Modified
- When the PR is Merged: All targets shall be built
This PR applies the New and Updated Rules defined in `arch.yml`:
- For "Arch: arm": Build `arm-01`, `arm-02`, ...
- For "Arch: arm64": Build `other`
- For "Arch: simulator": Build `sim-01`, `sim-02`
- For "Arch: x86_64": Build `other`
- For Simple PRs (One Arch Label + One Size Label): Skip the macOS and Windows builds (`macos`, `macos/sim-*`, `msys2`) since these builds are costly and slow
- Except for "Arch: Simulator", which will enable the macOS Builds for `sim-01` and `sim-02`
- If GitHub CLI Fails: Build all targets
The code is explained here: https://github.com/apache/nuttx/issues/13775
When building a branch like `releases/12.7`, the CI Workflow incorrectly checks out the `master` branch of `nuttx-apps`, instead of `releases/12.7`. This PR fixes a typo in `apps_ref`, to check out the correct branch.
This PR proposes to enhance the CI Workflow, to skip the unnecessary NuttX Builds. Currently, NuttX Devs wait for the CI Builds to complete across All Architectures (Arm32, Arm64, RISC-V, Xtensa), even though they modified a Single Architecture. With this PR, the CI Workflow will build only the Modified Architecture.
The solution uses the Arch Labels for PRs. We target only the Simple PRs: One Arch Label + One Size Label (e.g. "Arch: risc-v, Size: XS")
- If "Arch: arm / arm64" is the only non-size label, then we build `other`, `arm-01`, `arm-02`, ...
- If "Arch: risc-v" is the only non-size label, then build `risc-v-01`, `risc-v-02`
- If "Arch: xtensa" is the only non-size label, then build `xtensa-01`, `xtensa-02`
- The above rules apply when the PR is Created or Modified
- When the PR is Merged: All targets shall be built
The code is explained here: https://github.com/apache/nuttx/issues/13775
The simple improvement is designed to speed up compilation and reduce download errors on github and local.
Added a folder nxtmpdir for storing third-party packages
nuttxworkspace
|
|- nuttx
|- apps
|- nxtmpdir
tools/Unix.mk:
added export NXTMPDIR := $(WSDIR)/nxtmpdir
tools/configure.sh:
added option -S creates the nxtmpdir folder for third-party packages.
tools/Config.mk:
added macro
CLONE - Git clone repository.
CHECK_COMMITSHA - Check if the branch contains the commit SHA-1.
tools/testbuild.sh:
added option -S
For now I added in the folder this package
ESP_HAL_3RDPARTY_URL = https://github.com/espressif/esp-hal-3rdparty.git
ARCH
arch/xtensa/src/esp32/Make.defs
arch/xtensa/src/esp32s2/Make.defs
arch/xtensa/src/esp32s3/Make.defs
arch/risc-v/src/common/espressif/Make.defs
arch/risc-v/src/esp32c3-legacy/Make.defs
but you can also add other packages (maybe also of apps)
This PR modifies NuttX CI and GitHub Actions, to comply with ASF Policy. Right now, every NuttX Pull Request will trigger 24 Concurrent Jobs (GitHub Runners), executing them in parallel: https://lupyuen.github.io/articles/ci
According to ASF Policy: We should run at most 15 Concurrent Jobs: https://infra.apache.org/github-actions-policy.html
Thus we'll cut down the Concurrent Jobs from 24 down to 15. That's 12 Linux Jobs, 2 macOS, 1 Windows. (Each job takes 30 mins to 2 hours)
(1) Right now our "Linux > Strategy" is a flat list of 20 Linux Jobs, all executed in parallel
(2) We change "Linux > Strategy" to prioritise by Target Architecture, and limit to 12 concurrent jobs
(3) So NuttX CI will initially execute 12 Build Jobs across Arm32, Arm64, RISC-V, Simulator and Xtensa. As they complete, NuttX CI will execute the remaining 8 Build Jobs (for Arm32).
(4) This will extend the Overall Build Duration from [2 hours](https://github.com/apache/nuttx/actions/runs/10817443237) to [2.25 hours](https://github.com/lupyuen4/ci-nuttx/actions/runs/10828246630)
(5) We'll also limit macOS Jobs to 2, Windows Jobs to 1
Currently concerns only arm.
tools/ci/testlist/msys2.dat:
At the moment I only added the board nucleo-l152re:nsh
.github/workflows/build.yml:
Enabled cmake for msys2
tools/testbuild.sh
added option -N Use CMake with Ninja as the backend.
updated help.
added date to print startup board
added HOST=Msys
github/workflows/build.yml
added option -N job docker
function bloaty() Add -D CMAKE_INSTALL_PREFIX="${NUTTXTOOLS}"/bloaty.
Improves workflow execution time because it is now cached.
Changed reference file to calculate the hash for key of actions/cache@v4. Now it is darwin.sh.
the check isn't really enabled and enforce before
due to a mass of false alarm, but recently it break
ci frequently, so it's better to remove it now.
Signed-off-by: Xiang Xiao <xiaoxiang@xiaomi.com>
Clang relies on clang-extdef-mapping for ctu checking. But clang-arm doesn't provide this file
A tweaking way to support this feature is copy a `clang-extdef-mapping` from other clang version.
Signed-off-by: xinbingnan <xinbingnan@xiaomi.com>
.github/workflows/build.yml: added CodeChecker support for GitHub Workflow
tools/testbuild.sh: added support for CodeChecker checks
- Added support for CodeChecker checks.
- Generate inspection reports and summaries.
- After the task is executed, the logs are compressed and the database is packaged.
Signed-off-by: xinbingnan <xinbingnan@xiaomi.com>
Lup Yuen Lee