This PR enables the CI Builds for msys2, msvc, arm-08, arm-10, arm-13 for Complex PRs. We disable the CI Builds for arm-01, arm-09, arm-11.
This will help to fix the recent breakage of builds: https://github.com/apache/nuttx/issues/14598
This PR splits the CI Build Job sim-01 and adds sim-03:
Before the Split: Simulator Jobs take up to 1.5 hours to complete
- sim-01 (1 hour 31 mins): adb, citest, lvgl, matter
- sim-02 (28 mins): posix_test, sqlite
After the Split: Simulator Jobs will complete within 1 hour
- sim-01 (58 mins): adb, citest
- sim-02 (35 mins): lvgl, matter
- sim-03 (28 mins): posix_test, sqlite
This will help us comply with the ASF Policy for GitHub Actions, as explained here: https://github.com/apache/nuttx/issues/14376
Not all labels are added because there is no match with the label name.
issue_labeler.yml
os: Linux -> os: linux
001_bug_report.yml
Type: bug -> Type: Bug
Corrected label in links
002_feature_request.yml
Type: enhancement -> Type: Enhancement
Corrected label in links
003_help.yml
Type: question ->Type: Question
Corrected label in links
This PR updates the Build Rules `arch.yml` to build only these Arm32 Targets when we create or update a Complex PR:
- arm-01, arm-03, arm-05, arm-06, arm-07, arm-09, arm-11
No changes for Simple PRs (arm-01 to arm-14) and for Merging PRs (also arm-01 to arm-14).
This will improve our breadth of CI Checks across Arm32 Targets, as explained here: https://github.com/apache/nuttx/issues/14376
This PR creates the new CI Build Jobs `arm64-01` and `x86_64-01`. The new jobs will split and offload the Arm64 and x86_64 Build Targets from `other`. This will reduce our usage of GitHub Runners, to comply with the ASF Policy for GitHub Actions. (Recently we see more PRs for Arm64 and x86_64)
Before the Split: Simple PRs (One Arch and/or One Board) for Arm64 and x86_64 require almost 1 hour for CI Build
- `other` (57 mins): AVR, SPARC, x86, PinePhone, QEMU Arm64, QEMU x86_64
After the Split: Simple PRs for Arm64 and x86_64 will complete under 30 mins
- `other` (24 mins): AVR, SPARC, x86
- `arm64-01` (29 mins): PinePhone, QEMU Arm64
- `x86_64-01` (9 mins): QEMU x86_64
To skip more unnecessary builds: Our Build Rules `arch.yml` shall ignore the label "Area: Documentation", so that a Simple PR + Docs is still a Simple PR. Previously we experienced longer CI Build Times, just because we added docs to our Simple PR. (Now our PR shall be built exactly like a Simple PR)
The updated CI code is explained here: https://github.com/apache/nuttx/issues/13775
When we submit or update a Complex PR that affects All Architectures (Arm, RISC-V, Xtensa, etc): CI Workflow shall run only half the jobs for RISC-V, Xtensa and Simulator:
- `risc-v-01` to `03`
- `xtensa-01`
- `sim-01`
When the Complex PR is Merged: CI Workflow will still run all jobs for RISC-V, Xtensa and Simulator:
- `risc-v-01` to `06`
- `xtensa-01` to `02`
- `sim-01` to `02`
Simple PRs with One Single Arch / Board will build the same way as before:
- `risc-v-01` to `06`
- `xtensa-01` to `02`
- `sim-01` to `02`
We hope to lower drastically our usage of GitHub Runners before the ASF Deadline, as explained here: https://github.com/apache/nuttx/issues/14376
When we submit or update a Complex PR that affects All Architectures (Arm, RISC-V, Xtensa, etc): CI Workflow shall run only half the jobs. Previously CI Workflow will run `arm-01` to `arm-14`, now we will run only `arm-01` to `arm-07`.
When the Complex PR is Merged: CI Workflow will still run all jobs `arm-01` to `arm-14`
Simple PRs with One Single Arch / Board will build the same way as before: `arm-01` to `arm-14`
This is explained here: https://github.com/apache/nuttx/issues/14376
Note that this version of `arch.yml` has diverged from `nuttx-apps`, since we are unable to merge https://github.com/apache/nuttx/pull/14377
To avoid the following CI error:
This environment is externally managed
--> To install Python packages system-wide, try apt install
python3-xyz, where xyz is the package you are trying to
install.
If you wish to install a non-Debian-packaged Python package,
create a virtual environment using python3 -m venv path/to/venv.
Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
sure you have python3-full installed.
If you wish to install a non-Debian packaged Python application,
it may be easiest to use pipx install xyz, which will manage a
virtual environment for you. Make sure you have pipx installed.
See /usr/share/doc/python3.12/README.venv for more information.
This PR extends the CI Build Rules to eliminate more unnecessary builds. We use the Board Label for PRs:
- Previously: "Arch: arm" will build only `arm-01` to `arm-14`
- Now: "Board: arm" will also build `arm-01` to `arm-14`
- This applies only to Simple PRs: One Arch Label + One Board Label + One Size Label, like "Arch: arm, Board: arm, Size: L"
- If Arch Label and Board Label are both present: They must be the same. Otherwise All Targets shall be built.
- Works with Arm32, Arm64, RISC-V, Simulator, x86_64 and Xtensa
The updated code is explained here: https://github.com/apache/nuttx/issues/13775
To speed up the CI Workflow, this PR splits the CI Build Jobs for RISC-V into smaller jobs. Each job will now complete within 1 hour.
Before the PR: There are 2 jobs for RISC-V, each requiring more than 1.5 hours
- `risc-v-01` (1 hour 42 mins): BL602, Ox64, ESP32-C3 / C6 / H2
- `risc-v-02` (1 hour 41 mins): K230, Icicle, QEMU, RV32M1-Vega
After the PR: The build is spread across 6 jobs for RISC-V, each job completes within 1 hour
- `risc-v-01` (19 mins): BL602, Ox64
- `risc-v-02` (44 mins): ESP32-C3
- `risc-v-03` (45 mins): ESP32-C6, ESP32-H2
- `risc-v-04` (31 mins): K230, Icicle
- `risc-v-05` (41 mins): QEMU CITest
- `risc-v-06` (38 mins): Rest of QEMU, RV32M1-Vega
Following the same convention as the Arm32 Build Jobs, the above jobs are sorted by Target Name. Performance of the RISC-V Build Jobs is discussed in https://github.com/apache/nuttx/issues/13775
CI Build Job `arm-05` (runtime 2 hours) has become the Performance Bottleneck for CI Workflow. That's because `arm-05` builds too many targets for nRF, RP2040, SAM 3, SAM A and SAM D. This PR splits `arm-05` into multiple smaller jobs, to reduce the CI Build Duration.
Before the PR: `arm-05` is overloaded, build requires 2 hours
- `arm-05` (2 hours): nRF, RP2040, SAM 3, SAM A, SAM D
- `arm-06` (56 mins): STM32 [a-m]*
After the PR: `arm-05` is offloaded (to `arm-06` and `arm-07`), completes within 1 hour
- `arm-05` (47 mins): nRF
- `arm-06` (1 hour): Reserve for RP2040 exclusively
- `arm-07` (1 hour 15 mins): SAM 3, SAM A, SAM D, STM32 [a-m]*
Build Jobs are sorted by Target Name. So we cascade the changes and rename the Build Jobs: `arm-07` becomes `arm-08`, `arm-08` becomes `arm-09` etc. Then `arm-13` becomes a new job `arm-14`. (Which we added to `build.yml`)
Performance of `arm-05` is discussed in https://github.com/apache/nuttx/issues/13775 and https://github.com/apache/nuttx/issues/12773
This PR continues to enhance the CI Workflow, to skip the unnecessary NuttX Builds. The PR will update the CI Build Workflow `build.yml`, to call the Refactored Build Rules in `arch.yml` (which is a Reusable Workflow).
The original rules were migrated from `build.yml` to `arch.yml`:
- We target only the Simple PRs: One Arch Label + One Size Label (e.g. "Arch: risc-v, Size: XS")
- For "Arch: risc-v": Build `risc-v-01`, `risc-v-02`
- For "Arch: xtensa": Build `xtensa-01`, `xtensa-02`
- The above rules apply when the PR is Created or Modified
- When the PR is Merged: All targets shall be built
This PR applies the New and Updated Rules defined in `arch.yml`:
- For "Arch: arm": Build `arm-01`, `arm-02`, ...
- For "Arch: arm64": Build `other`
- For "Arch: simulator": Build `sim-01`, `sim-02`
- For "Arch: x86_64": Build `other`
- For Simple PRs (One Arch Label + One Size Label): Skip the macOS and Windows builds (`macos`, `macos/sim-*`, `msys2`) since these builds are costly and slow
- Except for "Arch: Simulator", which will enable the macOS Builds for `sim-01` and `sim-02`
- If GitHub CLI Fails: Build all targets
The code is explained here: https://github.com/apache/nuttx/issues/13775
This PR continues to enhance the CI Workflow, to skip the unnecessary NuttX Builds. The changes in this PR will not take effect until the next PR, which will switch `build.yml` to use the rules in this PR.
In this PR, we refactor the CI Build Rules into a separate Reusable Workflow `arch.yml`. The original rules were migrated to `arch.yml`:
- We target only the Simple PRs: One Arch Label + One Size Label (e.g. "Arch: risc-v, Size: XS")
- For "Arch: risc-v": Build `risc-v-01`, `risc-v-02`
- For "Arch: xtensa": Build `xtensa-01`, `xtensa-02`
- The above rules apply when the PR is Created or Modified
- When the PR is Merged: All targets shall be built
New and Updated Rules:
- For "Arch: arm": Build `arm-01`, `arm-02`, ...
- For "Arch: arm64": Build `other`
- For "Arch: simulator": Build `sim-01`, `sim-02`
- For "Arch: x86_64": Build `other`
- For Simple PRs (One Arch Label + One Size Label): Skip the macOS and Windows builds (`macos`, `macos/sim-*`, `msys2`) since these builds are costly and slow
- Except for "Arch: Simulator", which will enable the macOS Builds for `sim-01` and `sim-02`
- If GitHub CLI Fails: Build all targets
The code is explained here: https://github.com/apache/nuttx/issues/13775
When building a branch like `releases/12.7`, the CI Workflow incorrectly checks out the `master` branch of `nuttx-apps`, instead of `releases/12.7`. This PR fixes a typo in `apps_ref`, to check out the correct branch.
This PR proposes to enhance the CI Workflow, to skip the unnecessary NuttX Builds. Currently, NuttX Devs wait for the CI Builds to complete across All Architectures (Arm32, Arm64, RISC-V, Xtensa), even though they modified a Single Architecture. With this PR, the CI Workflow will build only the Modified Architecture.
The solution uses the Arch Labels for PRs. We target only the Simple PRs: One Arch Label + One Size Label (e.g. "Arch: risc-v, Size: XS")
- If "Arch: arm / arm64" is the only non-size label, then we build `other`, `arm-01`, `arm-02`, ...
- If "Arch: risc-v" is the only non-size label, then build `risc-v-01`, `risc-v-02`
- If "Arch: xtensa" is the only non-size label, then build `xtensa-01`, `xtensa-02`
- The above rules apply when the PR is Created or Modified
- When the PR is Merged: All targets shall be built
The code is explained here: https://github.com/apache/nuttx/issues/13775
This change adds a specifier to github lables created with "new issue":
- for labels related to architecture specifier is "Arch: xxx"
- for labels related to issue type specifier is "Type: xxx"
- for labels related to host OS type specifier is "OS: xxx"
- for labels related to issue area type specifier is "Area: xxx"
The simple improvement is designed to speed up compilation and reduce download errors on github and local.
Added a folder nxtmpdir for storing third-party packages
nuttxworkspace
|
|- nuttx
|- apps
|- nxtmpdir
tools/Unix.mk:
added export NXTMPDIR := $(WSDIR)/nxtmpdir
tools/configure.sh:
added option -S creates the nxtmpdir folder for third-party packages.
tools/Config.mk:
added macro
CLONE - Git clone repository.
CHECK_COMMITSHA - Check if the branch contains the commit SHA-1.
tools/testbuild.sh:
added option -S
For now I added in the folder this package
ESP_HAL_3RDPARTY_URL = https://github.com/espressif/esp-hal-3rdparty.git
ARCH
arch/xtensa/src/esp32/Make.defs
arch/xtensa/src/esp32s2/Make.defs
arch/xtensa/src/esp32s3/Make.defs
arch/risc-v/src/common/espressif/Make.defs
arch/risc-v/src/esp32c3-legacy/Make.defs
but you can also add other packages (maybe also of apps)
This PR modifies NuttX CI and GitHub Actions, to comply with ASF Policy. Right now, every NuttX Pull Request will trigger 24 Concurrent Jobs (GitHub Runners), executing them in parallel: https://lupyuen.github.io/articles/ci
According to ASF Policy: We should run at most 15 Concurrent Jobs: https://infra.apache.org/github-actions-policy.html
Thus we'll cut down the Concurrent Jobs from 24 down to 15. That's 12 Linux Jobs, 2 macOS, 1 Windows. (Each job takes 30 mins to 2 hours)
(1) Right now our "Linux > Strategy" is a flat list of 20 Linux Jobs, all executed in parallel
(2) We change "Linux > Strategy" to prioritise by Target Architecture, and limit to 12 concurrent jobs
(3) So NuttX CI will initially execute 12 Build Jobs across Arm32, Arm64, RISC-V, Simulator and Xtensa. As they complete, NuttX CI will execute the remaining 8 Build Jobs (for Arm32).
(4) This will extend the Overall Build Duration from [2 hours](https://github.com/apache/nuttx/actions/runs/10817443237) to [2.25 hours](https://github.com/lupyuen4/ci-nuttx/actions/runs/10828246630)
(5) We'll also limit macOS Jobs to 2, Windows Jobs to 1
Currently concerns only arm.
tools/ci/testlist/msys2.dat:
At the moment I only added the board nucleo-l152re:nsh
.github/workflows/build.yml:
Enabled cmake for msys2
Template
Bug report
Report a bug to improve NuttX stability
Feature request
Request an enhancement for NuttX
General Help
Get general support regarding NuttX
Action
An action for automatically labelling issues
tools/testbuild.sh
added option -N Use CMake with Ninja as the backend.
updated help.
added date to print startup board
added HOST=Msys
github/workflows/build.yml
added option -N job docker
function bloaty() Add -D CMAKE_INSTALL_PREFIX="${NUTTXTOOLS}"/bloaty.
Improves workflow execution time because it is now cached.
Changed reference file to calculate the hash for key of actions/cache@v4. Now it is darwin.sh.
the check isn't really enabled and enforce before
due to a mass of false alarm, but recently it break
ci frequently, so it's better to remove it now.
Signed-off-by: Xiang Xiao <xiaoxiang@xiaomi.com>
Lup Yuen Lee