Commit Graph

544 Commits

Author SHA1 Message Date
Marc Herbert ad8b54f095 xtensa-build-zephyr.py: add new PlatformConfig dataclass
Reduces duplication and provides more flexibility.

Also switch the list of platforms to a dict.

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2023-03-14 12:54:32 +00:00
Marc Herbert 8aab18351f xtensa-build-zephyr: fix DEFAULT_TOOLCHAIN_VARIANT spill on next platf
xtensa-build-zephyr.py has always defined XTENSA_ and other environment
variables in the current environment shared by all platforms. This was
always bad but apparently never a problem until the addition of the new
DEFAULT_TOOLCHAIN_VARIANT (xcc or clang) variable.

Before DEFAULT_TOOLCHAIN_VARIANT, each platform's environment would
simply override the previous one. However with the new
DEFAULT_TOOLCHAIN_VARIANT, the current environment has precedence for
more flexibility. This makes each platform "spill" onto the next one and
`xtensa-build-zephyr.py -p tgl mtl` fail like this:

```
-- Board: intel_adsp_ace15_mtpm
-- Found toolchain: xcc (/home/XCC/install/tools)
CMake Error at zephyr/cmake/compiler/xcc/generic.cmake:9 (message):
  Zephyr was unable to find the toolchain.  Is the environment
  misconfigured?

  User-configuration:

  ZEPHYR_TOOLCHAIN_VARIANT: xcc

  Internal variables:

  CROSS_COMPILE:
  /home/XCC/install/tools/RI-2022.10-linux/XtensaTools/bin/xt-
```

To fix this, stop modifying the current os.environ and use a new, fresh
os.environ.copy() for each platform instead.

Fixes commit 309fa264e2 ("xtensa-build-zephyr.py: upgraded Xtensa
toolchain for MTL")

History repeats itself: commit 6bedd8e742 ("xtensa-build-zephyr: fix
RIMAGE_KEY when building multiple platforms") fixed the same logical
error months ago in a different script.

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2023-03-13 14:19:49 +02:00
Guennadi Liakhovetski b637889efb platform: remove support for cAVS 2.0 platforms
Remove all support for cAVS 2.0 platformsm including Ice Lake and
Jasper Lake, they aren't supported any more.

Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
2023-03-07 14:02:18 +02:00
Iuliana Prodan 9c3bba8210 scripts: qemu-check.sh: update READY_IPC for imx8
Update READY_IPC value based on changes regarding
interrupt initialization.

Signed-off-by: Iuliana Prodan <iuliana.prodan@nxp.com>
2023-03-03 09:09:25 +02:00
Guennadi Liakhovetski dc9ba281d7 platform: remove support for cAVS 1.8 platforms
Remove all support for cAVS 1.8 platformsm including Cannon Lake,
Comet Lake, Whiskey Lake and Coffee Lake, they aren't supported
any more.

Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
2023-03-02 23:02:37 +00:00
Guennadi Liakhovetski 0f0acaae94 platform: remove support for cAVS 1.5 platforms
Remove all support for cAVS 1.5 platformsm including Apollo Lake,
Sky Lake, Kaby Lake, Broxton and Gemini Lake, they aren't supported
any more.

Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
2023-03-02 11:28:23 +00:00
Marc Herbert 37f27092be xtensa-build-platforms.py: build_rimage() only once at start
This will be required when signing is transferred to `west build`, see
https://github.com/zephyrproject-rtos/zephyr/pull/54700

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2023-03-01 15:22:13 +00:00
Marc Herbert a9680388e5 xtensa-build-zephyr.py: extract new build_rimage() function
Zero functional change. Next commit will build rimage _before_ building
the firmware which will be required when `west build` starts signing by
default (https://github.com/zephyrproject-rtos/zephyr/pull/54700)

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2023-03-01 15:22:13 +00:00
Andrey Borisovich 309fa264e2 xtensa-build-zephyr.py: upgraded Xtensa toolchain for MTL
Intel Meteorlake release toolchain is Xtensa RI-2022.10 version.
New toolchain does not support xt-xcc driver anymore so we are
forced to switch to new xt-clang driver.
To distinguish between default driver for the platform built,
(Xtensa GCC or Clang) added new field to script platform list
containing a toolchain name "xcc" or "xcc-clang" expected by
Zephyr project.

Signed-off-by: Andrey Borisovich <andrey.borisovich@intel.com>
2023-02-28 14:59:41 +00:00
Guennadi Liakhovetski 17fbf7de22 platform: remove support for Sue Creek
Remove all support for Sue Creek platforms, it isn't supported any
more.

Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
2023-02-20 15:46:16 +00:00
Guennadi Liakhovetski d80635054f platform: remove support for Broadwell and Haswell
Remove all support for Broadwell and Haswell platforms, they
aren't supported any more.

Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
2023-02-16 16:19:36 +00:00
Guennadi Liakhovetski 89f7aa0abd platform: remove support for Baytrail and Cherrytrail
Remove all support for Baytrail and Cherrytrail platforms, they
aren't supported on the "main" branch any more. To build SOF for them
use the "table-v2.2" branch.

Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
2023-02-08 13:36:04 +00:00
Andy Ross 3f5c0baba1 scripts: Add fuzz test wrapper
Simple script to explain how fuzz testing works

Signed-off-by: Andy Ross <andyross@google.com>
2023-02-08 11:47:56 +02:00
Marc Herbert a5dfa5669e xtensa-build-zephyr.py: checksum dsp_basefw.bin
When using --fw-naming AVS the .ri file is renamed to `dsp_basefw.bin`.

Also explain why it's still useful to identify non-deterministic images.

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2023-01-26 20:24:14 +02:00
Marc Herbert f85320ba8d scripts/xtensa-build-zephyr.py: checksum [sof_]version.h files
`git describe` differences happen, especially when trying to optimize
cloning and this is for instance what happened in
https://github.com/thesofproject/sof/pull/6950#issuecomment-1396150533
where zephyr tags were fetched on Linux but not on Windows

This also makes plain git version differences more obvious; no need to
scroll all the way up and scan the build logs.

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2023-01-26 20:24:14 +02:00
Marc Herbert cc00c57057 xtensa-build-zephyr.py: require zephyr.strip
Make CONFIG_BUILD_OUTPUT_STRIPPED mandatory so we can always compare
builds. It makes practically zero build space and time difference and
has huge reproductibility value, see discussion in
https://github.com/zephyrproject-rtos/zephyr/pull/51954

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2023-01-23 19:28:51 +02:00
Marc Herbert b4edab69de Revert "cmake: make sure sha1 is computed without filters"
Commit 3315681c69 ("cmake: make sure sha1 is computed without
filters") added the `--no-filters` option to the `git hash-object`
invocation used to produce a `src/` checksum embedded in all firmware
binaries and .ldc files.

This fixed a small mismatch for the (very few) people using the
_asymmetric_ `autocrlf = input`. However it created a huge mismatch for
the numerous people using the _symetric_ `autocrlf = true`.

So revert that commit, remove `--no-filters` and replace it with a build
time warning.

`--no-filters` produces a checksum of the files as they are on the
filesystem, as opposed to how they would be input into git after
optional CRLF -> LF conversion.

When `--no-filters` was added, no one was compiling in Windows. Most
developers were not performing any conversion because there is no
`autocrlf` conversion by default on Linux.  Files were identical inside
and outside of git and life was simple.

Simple life except for at least one indomitable Linux developer who had
the _asymmetric_ `autocrlf = input` setting. Also, the SOF git repo
always had two exceptional (and generated) files with CRLF end of lines;
see issue #5917 for details (all other files have LF end of lines). So
for that "asymmetric" developer only and these two files only, `autocrlf
= input` converted these 2 files to LF before hashing and this caused a
different checksum. `--no-filters` stopped that conversion and "solved"
that mismatch.

But now Windows has entered the stage. On Windows, the default is
usually (and unfortunately) the symmetric `autocrlf = true`. It is the
default for Github's Windows runner. This symmetric default converts ALL
other files to CRLF when extracting them out of git. This causes `git
hash-object --no-filters` to produce a _different hash for all but 2
files on Windows_!

The only solution is to:

1. Drop the `--no-filters` to align everything on what is _stored in
   git_. What is stored in git is the only thing we are sure all users have
   in common.

2. Request users to use only _symmetric_ `autocrlf` settings so any
   optional input+output conversion cancels itself out.

3. Convert odd files and make sure they all have LF end of lines.

4. In the future, drop this entire git hash-object technique which is
   flawed in multiple ways and has been causing multiple issues. Much,
   much more work than this very small revert though.

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2023-01-18 12:17:51 +00:00
Marc Herbert 75fa04de09 scripts/sudo-cwd.sh: don't try to copy missing sudoers.d/ permissions
Copying a file that does not exist obviously fails.

This bug was found when trying to switch from the current "Developer
Image" to the smaller "CI Image":
https://github.com/zephyrproject-rtos/docker-image/pkgs/container/zephyr-build

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2023-01-06 14:52:09 +00:00
Marc Herbert 2b85c2c358 scripts: fix check whether CMake configuration succeeded
To test whether CMake configuration can be skipped, look for either a
ninja.build or Makefile.

An existing build directory is NOT proof that CMake configuration
succeeded. When CMake configuration failed, trying to build again fails
with a super confusing "Makefile not found" error message.

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2023-01-04 16:09:08 +00:00
Marc Herbert c6e7011e1e xtensa-build-zephyr.py: convert sof-info/ text files from CRLF to LF
Building on Windows generates CRLF text files. Standardize checksums to
LF files.

As usual, files in the build directory are left untouched.

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2022-12-23 11:02:21 +00:00
Marc Herbert e75fcd1b45 xtensa-build-zephyr.py: search "strip" in string, not in pathlib
Fixes pylint warning "E1135 (unsupported-membership-test)". Not sure
this warning is valid because the code worked anyway but it does not
hurt.

Fixes commit bc394916d0 ("xtensa-build-zephyr.py: switch type of
InstFile() names to pathlib")

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2022-12-23 11:02:21 +00:00
Andrey Borisovich b34cb8bfbb xtensa-build-zephyr.py: change install tree print to ASCII style
During testing of Windows CI workflow on windows-latest turned out
that github action console has issues rendering python
anytree.RenderTree graph with default style of
anytree.render.ContStyle (continuous vertical and horizontal lines).
Tests on local machines in cmd, pwershell and msys shells
proves, that all shells can render the graph (while powershell
does not do this correctly and graph is malformed).
Changing graph style to ASCII resolves the issue with Github actions.

Signed-off-by: Andrey Borisovich <andrey.borisovich@intel.com>
2022-12-21 11:46:28 +00:00
Marc Herbert 9b2453e236 xtensa-build-zephyr.py: compress sof-info files in parallel
On my system this brings the install_platform() duration down from about
3 seconds to less than 1 second.

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2022-12-20 15:44:20 +00:00
Marc Herbert 4ac3e3ff44 xtensa-build-zephyr.py: don't leave the install of an old build behind
In theory a failure from this script should immediately stop the show
and no one should then use the staging directory.

In practice we cannot make sure, especially not with shell scripts. So
make sure no previous build of a requested platform is left behind.

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2022-12-19 17:25:28 +00:00
Marc Herbert 424da2cd9d zephyr/docker: pass http[s]_proxy variables to the container
Support downloads from within the container.

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2022-12-16 15:19:44 -08:00
Marc Herbert 906508bd79 xtensa-build-zephyr.py: checksum deterministic configs.c
.config is not deterministic because it has absolute paths in comments
and its order seems hard to predict. configs.c has the same information
generated in a determistic way.

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2022-12-16 13:42:08 +02:00
Marc Herbert bc394916d0 xtensa-build-zephyr.py: switch type of InstFile() names to pathlib
Make sure any forward slashes in strings are immediately converted to
OS-dependent directory separators.

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2022-12-16 13:42:08 +02:00
Tinghan Shen f28dc601cb platform: Add mt8188 to Kconfig
Add mt8188 to Kconfig.

Signed-off-by: Tinghan Shen <tinghan.shen@mediatek.com>
2022-12-15 16:11:23 +00:00
Marc Herbert 501204c684 xtensa-build-zephyr.py: objcopy --remove-section .comment zephyr.strip
Make stripped ELF files compiled by XCC identical across different
machines.

Some Xtensa compilers (ab?)use the .ident / .comment
section and append the typically absolute and not
reproducible /path/to/the.c file after the usual
compiler ID.

https://sourceware.org/binutils/docs/as/Ident.html

strip --strip-all does not remove the .comment section.
Remove ourselves like some gcc test scripts do:

 https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=c7046906c3ae

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2022-12-15 13:55:07 +02:00
Marc Herbert 6516f907aa sparse: pass platform argument to error filter script
So we can make adjust the warnings based on the platform.

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2022-12-12 12:05:26 +00:00
Marc Herbert e39205cc11 parse_sparse_output.sh: new sparse_errors array
Zero functional change, pure preparation for the next commit.

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2022-12-12 12:05:26 +00:00
Marc Herbert 09386bc0d9 Revert "xtensa-build-zephyr.py: west update --depth 5 --narrow"
This reverts commit ce28e09bd3.

This fixes Zephyr's git describe command and build reproducibility.

I tried fairly hard various git fetch options like --shallow-exclude
and --shallow-since but they did not save that much download (200MB at
best), required some hardcoding and most importantly they make complete
clones shallow again when invoked unconditionally. Not worth the
effort, build reproducibility is more important.

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2022-12-09 15:49:28 +00:00
Marc Herbert 280dca92b5 xtensa-build-zephyr.py: build rimage with -GNinja by default
This saves a couple seconds when building from scratch on Linux.

On Linux the default CMake generator is "Makefiles" which is _not_
parallel by default.

Thanks to the previous commit it's still possible to manually switch to
"Makefiles" if desired.

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2022-12-07 17:21:43 +00:00
Marc Herbert 2fcab330b9 xtensa-build-zephyr.py: do not CMake-reconfigure rimage every time
It's pointless and hardcodes the CMake generator.

Also remove wrong comment added in commit
6cba64d2cb ("xtensa-build-zephyr.py: fix a few minor pylint warnings")
The rimage part of the comment was flat out wrong.
The smex part of the comment is correct but in the wrong place.

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2022-12-07 17:21:43 +00:00
Iuliana Prodan 6131901c46 scripts: qemu-check.sh: update READY_IPC for imx8
Update READY_IPC value based on changes regarding MU reset.
READY_IPC value comes from:
- clear GP pending interrupt #0 and #1 from MU's xSR register;
- enable GP #0 and #1 for Host -> DSP and DSP -> Host
message notification from MU's xCR register;
- now interrupt host to tell it we are done booting
by setting GIRn bit in MU's xCR register.

So, "00 00 00 c0 00 00 04 c0" is the MU's xSR and xCR registers:
xSR: c0000000 and xCR: c0040000

Signed-off-by: Iuliana Prodan <iuliana.prodan@nxp.com>
2022-12-05 14:33:46 +00:00
Marc Herbert 9fa1a99d8b xtensa-build-zephyr.py: install and checksum .elf, .lst and other files
Sample new output below. The sha256 is computed on the _uncompressed_
files and _not_ affected by gzip metadata.

```
build-sof-staging
├── sof
│   ├── community
│   │   ├── sof-imx8.ri         sha256=50c5423d2355ef3ed91...
│   │   └── sof-tgl.ri          sha256=6b1d26c12a63de5dfc8...
│   ├── sof-imx8.ldc    sha256=520d365d188d5ef4e907d3dd8c9...
│   └── sof-tgl.ldc     sha256=cdce6ad340d4dc047e71e738eb0...
├── sof-info
│   ├── imx8
│   │   ├── config.gz
│   │   ├── zephyr.elf.gz
│   │   ├── zephyr.lst.gz       sha256=8c83c3fc92df0a871dd...
│   │   ├── zephyr.map.gz
│   │   └── zephyr.strip.gz     sha256=d3ce3d3450c67bb3580...
│   └── tgl
│       ├── boot.mod.gz         sha256=d9c9e82e75fa6d061bf...
│       ├── config.gz
│       ├── main.mod.gz
│       ├── stripped-main.mod.gz        sha256=c367dccca6d...
│       ├── zephyr.elf.gz
│       ├── zephyr.lst.gz       sha256=5474bc5e58a5d000109...
│       ├── zephyr.map.gz
│       └── zephyr.strip.gz     sha256=6285f41c0682b33b7e0...
└── tools
    ├── cavstool.py
    ├── cavstool_client.py
    ├── cavstwist.sh
    ├── mtrace-reader.py
    ├── remote-fw-service.py
    └── sof-logger
```

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2022-11-30 11:59:33 +00:00
Marc Herbert 94f93c21dc xtensa-build-zephyr.py: indent new install_platform() function
Pure whitespace change.

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2022-11-30 11:59:33 +00:00
Marc Herbert 4f98441d3c xtensa-build-zephyr.py: extract new function install_platform()
Zero functional change.

Lowers the score of the following pylint warnings:

- Too many local variables (38/15) (too-many-locals)  41->38
- Too many branches (26/12) (too-many-branches)       29->26
- Too many statements (116/50)(too-many-statements)  127->116

Keep existing indentation for now to help git blame -Mnn. Will fix in next
commit.

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2022-11-30 11:59:33 +00:00
Marc Herbert 61fe7da105 xtensa-build-zephyr.py: fix misleading -i IPC4 help string
-i IPC4 is _not just_ an overlay change, there's an rimage change too.

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2022-11-30 11:59:33 +00:00
Marc Herbert d70ac67032 parse_sparse_output.sh: ignore skipped topology2 error
Off-topic here.

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2022-11-22 09:19:45 +00:00
Marc Herbert 842c39fb0f xtensa-build-zephyr.py: add missing RI_INFO_UNSUPPORTED platforms.
Copy the entire list of UNSIGNED_RI platforms from
src/arch/xtensa/CMakeLists.txt, adding the ones that don't use Zephyr
yet.

Also rename NO_RIMAGE_PLATFORMS to RI_INFO_UNSUPPORTED not to give the
wrong impression that imx8* don't use rimage.

Fixes commit 3a9413eebd ("xtensa-build-zephyr.py: restore lost
reproducible .ri checksum")

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2022-11-17 13:54:54 +00:00
Marc Herbert c97493b37a xtensa-build-zephyr.py: detect multiple rimage checkouts and abort
When cloning sof under a different directory name like
sof-experiment1 ((which is bad but we can't stop it unfortunately and it
keeps happening), it's common to end up with two rimage git checkouts:
sof/rimage and sof-experiment1/rimage. Detect this and fail immediately.

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2022-11-09 14:15:07 +00:00
Marc Herbert d95004a62e xtensa-build-zephyr.py: trivial re-use of rimage_source_dir variable
Removes some minor duplication.

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2022-11-09 14:15:07 +00:00
Marc Herbert b7708182fb .github: fail on errors and address space warnings in sparse output
sparse does not have a useful exit status so let's use 'grep' instead.

Fixes: #6317

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2022-11-01 10:55:58 +00:00
Marc Herbert e0722f493a xtensa-build-zephyr.py: de-hardcode mode=511 in mkdir()
These hardcoded 511 permissions came with the initial commit
1de3ef3675 ("Rewritten xtensa-build-zephyr.sh to python") without any
particular explanation.

On Unix, permissions of new files and directories are a user preference
set with the `umask`, different applications are not supposed to create
directories differently.

On Windows/NTFS it's not clear what 511 maps to.

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2022-10-27 21:38:56 +01:00
Marc Herbert 3a9413eebd xtensa-build-zephyr.py: restore lost reproducible .ri checksum
At the end of the build, parse the .ri file and print a reproducible
checksum of the its content.

This feature was in xtensa-build-zephyr.sh but never made it to the .py
conversion. It was lost when the .sh file was deleted in commit
ef028f634a ("Removed obsolete xtensa-build-zephyr.sh")

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2022-10-24 16:15:12 +01:00
Marc Herbert ce28e09bd3 xtensa-build-zephyr.py: west update --depth 5 --narrow
Restore previous optimization. Can easily be undone with
git -C zephyr/ fetch --unshallow.

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2022-10-24 12:10:45 +01:00
Marc Herbert 9a3a518644 xtensa-build-zephyr.py: pass -j option to west -> ninja
This is especially useful when compiling each C file requires a
network round-trip to a license server

Also stop overriding the ninja default when building rimage.

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2022-10-24 12:10:45 +01:00
Marc Herbert adce544450 xtensa-build-zephyr.py: simplify installation, prepare for more files
Simplify the final installation code to prepare for installing more
files. Zero functional change.

Notably perform the cavs naming trick inside the build directory first
which allows using a destination directory instead of a destination file.

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2022-10-21 20:02:11 +03:00
Marc Herbert ab7c1a2c7b xtensa-build-zephyr.py: de-interleave signing and installation code
Very minor, zero-change re-ordering to make the next commit easier to
review. Fixes commit 762f63bf8b ("xtensa-build-zephyr: add option to
build FW in cavs naming style")

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2022-10-21 20:02:11 +03:00
Marc Herbert a42592eda1 xtensa-build-zephyr.py: don't make "." a special case in tree listing
Small simplification.

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2022-10-21 20:02:11 +03:00
Marc Herbert 0497212dca xtensa-build-zephyr.py: sort glob(**/*) list used for tree display
`pathlib.glob()` (and filesystems in general) provide no ordering
guarantee, which is likely why every glob() example in
https://docs.python.org/3/library/pathlib.html#pathlib.Path.glob
is sorted(). Follow these examples and use `sorted()`.

Fix error handling to run _before_ the `list index out of range`
exception hits. Also make it fatal now because there's no more any good
reason for this to fail.

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2022-10-21 20:02:11 +03:00
Marc Herbert 6cba64d2cb xtensa-build-zephyr.py: fix a few minor pylint warnings
Declare globals at the top level.
Stop shadowing globals.
Replace `== or ==` with `in [ ]`.

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2022-10-21 20:02:11 +03:00
Marc Herbert b5abf6c8f3 xtensa-build-zephyr: disable pylint bad-whitespace, module-doc, global
Also re-organize the header.

Absolutely zero functional change.

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2022-10-21 20:02:11 +03:00
Marc Herbert d155062195 scripts/build-tools.sh: switch to incremental build by default
Faster build and focused and readable logs at last.

Add a warning not to surprise anyone.

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2022-10-10 11:34:11 +01:00
Marc Herbert d56d8ab01c scripts/build-tools.sh: switch to Ninja because make is too verbose
Make prints a line for every topology that is already up to date and not
rebuilt. It's impossible to see what was done.

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2022-10-10 11:34:11 +01:00
Marc Herbert c28400baaa scripts/docker-run.sh: fix wrong uid in warning 1001 -> 1000
Fixes commit d09844ab98 ("zephyr/docker-build.sh: match UID with
'adduser' instead of 'chgrp -R'")

Also clarify comment and add reference to new sudo-cwd.sh script.

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2022-10-10 11:29:05 +01:00
Marc Herbert 0a4b1d62d5 sudo-cwd.sh: drop the all 'sof_' prefixes and references
This script is now generic. This was not done earlier to be gentle on
git blame.

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2022-10-10 11:29:05 +01:00
Marc Herbert 027be98371 zephyr/docker: move exec_as_sof_uid() to new sudo-cwd.sh
Besides making things more obvious, the important functional change is
that the user switch is now performed for _every_ invoked, command, not
just for the build command.

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2022-10-10 11:29:05 +01:00
Marc Herbert 5694cfa012 scripts: add link to CMake envvars FAQ next to XTENSA_SYSTEM definition
Environment variables like XTENSA_SYSTEM are an absolute CMake
pain. Add a link to the CMake FAQ and a one-line description why.

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2022-09-26 12:33:56 +01:00
Marc Herbert b19cd3f080 xtensa-build-zephyr.py: don't complain about cmake_args when pristine
Don't ask the user to delete the build directory when they are using the
option that deletes the build directory.

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2022-09-26 12:33:56 +01:00
Marc Herbert 9462baf8d4 xtensa-build-zephyr.py: remove misleading -i IPC3 option
`./xtensa-build-zephyr.py -i IPC3 mtl` builds the MTL default: IPC4.
This is wrong, remove the misleading "IPC3" option which never did
anything at all.

Fix the --help string to describe what actually happens. This is a build
script, no need for fancy abstractions and indirections that don't even
match reality. Just tell it like it is.

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2022-09-23 16:29:48 +01:00
Adrian Warecki 784bce763c mtl: Added meteorlake platform to the build system
Added new mtl platform to xtensa-build-zephyr.py.
Added ace directories to cmake files.
Added ACE to kconfig.
Add Meteorlake platform to be built with Zephyr under
CONFIG_ACE_VERSION_1_5 flag.

Signed-off-by: Andrey Borisovich <andrey.borisovich@intel.com>
Signed-off-by: Marcin Szkudlinski <marcin.szkudlinski@intel.com>
Signed-off-by: Adrian Warecki <adrian.warecki@intel.com>
Signed-off-by: Konrad Leszczynski <konrad.leszczynski@intel.com>
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Signed-off-by: Tomasz Leman <tomasz.m.leman@intel.com>
Signed-off-by: Rafal Redzimski <rafal.f.redzimski@intel.com>
Signed-off-by: Adrian Bonislawski <adrian.bonislawski@intel.com>
2022-09-21 15:15:26 +02:00
Damian Nikodem a823958a8d xtensa-build-zephyr: pass sof build version to rimage
Rimage expects to receive fw_ver_build to prepare the manifest.

Signed-off-by: Damian Nikodem <damian.nikodem@intel.com>
2022-09-15 09:19:16 +02:00
Fred Oh 8136e514bf host-testbench.sh: add FullTest variable and set 0 by default
FullTest variable is by default 0. When fulltest is required,
set the variable and run the script like,

$ FullTest=1 ./scripts/host-testbench.sh

Signed-off-by: Fred Oh <fred.oh@linux.intel.com>
2022-09-07 22:09:33 +01:00
Fred Oh 6b744bcc63 host-testbench.sh: use process_test.m for 8 components
Expand to more components with process_test. Components for test are
volume, eq-iir, eq-fir, dcblock, drc, multiband-drc, src and tdfb.

Signed-off-by: Fred Oh <fred.oh@linux.intel.com>
2022-09-07 22:09:33 +01:00
Keqiao Zhang 0e4208d85b xtensa-build-zephyr.py: copy mtrace-reader.py to tools folder
we will switch to using this new logging tool, copy it to the tools
folder, then it will be uploaded to CI storage after building.

Signed-off-by: Keqiao Zhang <keqiao.zhang@intel.com>
2022-09-07 13:17:54 +01:00
Marc Herbert 9e546b4b84 scripts/test-repro-build.sh: leave only TGL
Other platforms are not supported anymore in the main branch.

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2022-09-02 18:30:02 +01:00
Marc Herbert dfc6b46e1d scripts/docker-run.sh: fix wrong "id -n" command -> "id -u"
Fixes commit d09844ab98 ("zephyr/docker-build.sh: match UID with
'adduser' instead of 'chgrp -R'")

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2022-09-02 18:28:30 +01:00
Marc Herbert 8fee9e1306 xtensa-build-all.sh: remove byt..jsl platforms from '-a'
These platforms are not supported in the main branch anymore.

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2022-08-31 21:03:16 +01:00
Marc Herbert 8328cc50a9 xtensa-build-all.sh: add rmb to default platforms built with `-a`
The rmb toolchain was added in commit 4e85dfc95f ("docker: add
AMD/Rembrandt gcc toolchain") and before.

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2022-08-31 21:03:16 +01:00
Marc Herbert 9915d1e1fd cmake: stop sneakily downloading missing submodules at build time
Note this change does NOT affect Zephyr builds in any shape or form as
Zephyr builds simply don't use the CMake files changed by this commit.

Downloading missing submodules at build time was never a good idea;
always a hack. Downloading and building should always be kept separate
from each other for version control and bill of materials reasons and to
support "off-line" builds; build inputs should always be available.

Now that Zephyr builds have just moved away from git submodules
(replaced by west), stop sneakily downloading missing submodules at
build time while the user cannot notice, overwhelmed by the volume of
build logs. Someone building XTOS first and Zephyr second could
unknowningly end up in a "hybrid" and undesired situation where git
submodules and west would be BOTH pointing at the same rimage clone.

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2022-08-31 10:40:04 +01:00
Marc Herbert 44a5200c87 xtensa-build-zephyr: fix pylint "Redefine 'platform' from outer scope"
Fixes pylint warning `scripts/xtensa-build-zephyr.py:426:5: W0621:
Redefining name 'platform' from outer scope (line 14)
(redefined-outer-name)` and reduce "grep confusion": the script uses
`platform` A LOT.

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2022-08-19 09:45:14 +01:00
Marc Herbert 730e861505 xtensa-build-zephyr.py: pylint:disable=bad-indentation W0311
There's just too many of them and fixing them would cause a lot of churn
at a wrong time (big migration to west happening now)

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2022-08-19 09:45:14 +01:00
Marc Herbert b94a5fbc54 xtensa-build-zephyr.py: fix two minor pylint f-string warnings
Fix two W1309 f-string-without-interpolation warnings

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2022-08-19 09:45:14 +01:00
Marc Herbert 3c6dcdb574 xtensa-build-zephyr.py: abort when python version < 3.8
shlex.join() requires 3.8, maybe others too. 3.8 is 3 years old and it's
the default Python version in Ubuntu 20.04

Fixes #6121

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2022-08-19 09:45:14 +01:00
Andrey Borisovich ecbbd1b5d1 scripts: xtensa-build-zephyr.py fix for windows
West tool returns on windows OS different white characters
than on Linux OS. Paths outputted from west tool had been
stripped for any trailing whitespaces.
Fixes "xtensa-build-zephyr.py: use west manifest"
(608833cf27)

Signed-off-by: Andrey Borisovich <andrey.borisovich@intel.com>
2022-08-12 15:02:18 +01:00
Fred Oh 4e85dfc95f docker: add AMD/Rembrandt gcc toolchain
Add AMD/Rembrandt (rmb) gcc toolchain to thesofproject/sof docker image.

Signed-off-by: Fred Oh <fred.oh@linux.intel.com>
2022-08-10 21:25:50 +01:00
Marc Herbert 473ec05b3b xtensa-build-zephyr.py: abort immediately on `west build` failure
Stop immediately when `west build` fails. Do no keep going and show a
totally unrelated smex build failure.

Also fix error message when zephyr/ is missing.

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2022-08-05 15:53:50 +01:00
Marc Herbert b29242a2ad xtensa-build-zephyr.py: -p ristine deletes STAGING_DIR
Make the new -p pristine build option also delete the staging_dir. This
solves a problem where the following sequence was leaving a stale
platform1 build in staging:

xtensa-build-zephyr.py -p platform1
   < code >
xtensa-build-zephyr.py -p platform2

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2022-08-05 15:53:50 +01:00
Andrey Borisovich 70406a0db0 scripts: added versioning to xtensa-build-zephyr.py and west manifest
Added versioning to scripts/xtensa-build-zephyr.py to get version
information when incompatible changes are done to the script.
Added yml schema version number to west.yml manifest.

Signed-off-by: Andrey Borisovich <andrey.borisovich@intel.com>
2022-08-03 11:43:22 +01:00
Andrey Borisovich 4be98e4dc9 scripts: xtensa-build-zephyr.py removed calls over symlink
xtensa-build-zephyr.py had been using some execute_command
calls when building rimage and using rimage keys over
symlink from modules/audio/sof. Changed it so that calls are
executed now over normal sof directory and symlink is not needed
for script execution.

Signed-off-by: Andrey Borisovich <andrey.borisovich@intel.com>
2022-08-03 11:43:22 +01:00
Andrey Borisovich 627e29f287 scripts: xtensa-build-zephyr.py west reinitialization
Added new flag for non-interactive mode that should be used
when script is invoked programmatically.
In default (interactive) mode user is asked using a prompt whether to
reinitialize west manifest if it is initialized to manifest other
than SOFs.

Signed-off-by: Andrey Borisovich <andrey.borisovich@intel.com>
2022-08-03 11:43:22 +01:00
Andrey Borisovich 608833cf27 xtensa-build-zephyr.py: use west manifest
Added usage of west manifest in the script to update both
SOF and zephyr dependencies.
Renamed "clone-mode" to "update" flag - the logic where
sof dependencies and zephyr with dependencies is cloned.
Removed "point-mode" from script as it will no longer work
(west manifests require fixed directory structure).
Removed "zephyr revision" or "-z" argument as it is no longer
needed (now revision may be set in west.yml manifest).
Added "-p" pristine flag to rebuild platforms while removing build
directory.

Co-developed-by: Krzysztof Frydryk <krzysztofx.frydryk@intel.com>
Signed-off-by: Krzysztof Frydryk <krzysztofx.frydryk@intel.com>
Signed-off-by: Andrey Borisovich <andrey.borisovich@intel.com>
2022-08-03 11:43:22 +01:00
Andrey Borisovich ef028f634a Removed obsolete xtensa-build-zephyr.sh
This script had been rewritten to python xtensa-build-zephyr.py
script and is no longer used in CI.
Due to changes to sof processes related to Zephyr integration
this script will no longer be maintained.

Signed-off-by: Andrey Borisovich <andrey.borisovich@intel.com>
2022-07-19 09:51:29 +01:00
Anas Nashif afd28fb087 app: zephyr: remove overlays and adapt script
Remove overlays as they are the same board configurations available in
app/.
Keep common overlays and change script to reflect those changes.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
2022-07-18 14:22:01 +01:00
Marc Herbert 33f27d3477 xtensa-build-zephyr.py: speed up thanks to an _actually_ shallow clone
This will significantly speed up CI in some cases.

Fixes commit ffdf001eee ("xtensa-build-zephyr.py: clone zephyr with
--depth 5")

When I added --depth 5 to git clone in that commit I naively assumed git
fetch would "inherit" that. I think it does but only when fetching an
_existing_ (and shallow) branch, not when fetching something new.

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2022-07-11 09:22:32 +01:00
Fred Oh 9fc3833a5d docker: dump git log to a file before deleting projects
To track the docker images, need to keep all the hashes of the projects
builts in the image. Log git tree information to
/home/sof/work/sof_git_hash.txt in the docker image.

Signed-off-by: Fred Oh <fred.oh@linux.intel.com>
2022-07-08 21:27:27 +01:00
Fred Oh 85d68d0921 docker: change indentation and add HOME variable
No functionality change. Add some logs and minor clean up.

Signed-off-by: Fred Oh <fred.oh@linux.intel.com>
2022-07-08 21:27:27 +01:00
Xiaoyun Wu(Iris) 1e467a106f xtensa-build-zephyr: Auto pamam completion for --fw-naming
There are some restrictions when --fw-naming is set to 'AVS' for the
sake of providing reasonable FW outputs that make sense.

Restriction 1: When argument fw_naming is set as AVS, IPC4 ipc version
will be automatically chosen.

This is because --fw-naming 'AVS' is added for supporting IPC4 SOF build by
nature.

Restriction 2: When argument fw_naming is set as AVS, using platform
subdirectories will be automatically chosen.

The reason is that the platform subdirectories actually belong to the
"AVS" naming scheme, we must enable using platform subdirectories when
building FW in AVS naming convention.

This also ensure that we can tell firmwares of different platforms apart
by organizing them in different subdirectories.

Signed-off-by: Xiaoyun Wu(Iris) <xiaoyun.wu@intel.com>
2022-07-06 22:05:02 +01:00
Iris Wu 762f63bf8b xtensa-build-zephyr: add option to build FW in cavs naming style
Add argument option --fw-naming for specifying FW output with its
filename set with AVS style(filename as dsp_basefw.bin specifically)
OR SOF style(filename as sof-{platform}.ri). SOF style is by default.

This will be useful when building SOF+Zephyr IPC4 FW, because
dsp_basefw.bin will be loaded by default when snd_sof_pci parameter
ipc_type is set to 1 (which indicates running IPC4 mode).

Signed-off-by: Iris Wu <xiaoyun.wu@intel.com>
2022-07-06 22:05:02 +01:00
Marc Herbert 7c5bf651f8 xtensa-build-zephyr: copy ALL cavstool*.py scripts
Quoting https://github.com/zephyrproject-rtos/zephyr/pull/46880#pullrequestreview-1020096140

> No complaints about the code here. Just a general comment:

> I do worry that as cavstool grows features (first to add the test
> integration support, now the new client/server architecture) that
> we're obscuring the core code that does the loading behind a big
> facade of other stuff that also has to be run In Just The Right Way to
> make things work. That impedes development flow. It might be time to
> split the code up so that a minimal "just run this file please" tool
> can continue to be maintained.`

Long story short: `cavstool.py` is likely to be split in the future.

To be readi-er, copy the entire `zephyr/soc/xtensa/intel_adsp/tools/`
directory.

Copying the entire directory also copies scripts that will never be used
on a test device like `cavstool_client.py`. They're small Python script
sso we don't care: keep it simple.

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2022-07-04 15:56:34 +01:00
Jaska Uimonen b0d71a23a6 xtensa-build-zephyr.py: add tgl-h ipc4 overlay
Add missing IPC4 overlay target and toml config for tgl-h.

Signed-off-by: Jaska Uimonen <jaska.uimonen@intel.com>
2022-06-28 11:11:53 +01:00
Pierre-Louis Bossart 3315681c69 cmake: make sure sha1 is computed without filters
Local filters in ~/gitconfig, such as

[core]
	autocrlf = input

can impact the result of git hash-object. Make sure no filters are
used so that the hash value remains unmodified across user setups.

BugLink: https://github.com/thesofproject/sof/issues/5917
Signed-off-by: Pierre-Louis Bossart <pierre-louis.bossart@linux.intel.com>
2022-06-16 20:17:27 +01:00
Balakishorepati c7fc085bd5 scripts: add rembrandt support
Add rembrandt build support.

Signed-off-by: Balakishorepati <balaKishore.pati@amd.com>
2022-06-15 14:52:59 +01:00
Marc Herbert 9d2e82adf0 xtensa-build-zephyr.py: fix find_west_workspace() to return west topdir
When neither -c nor -p is passed, the script defaults to '-p
SOF_TOP/zephyrproject', initializing the `west_top` global with
that. When everything is already cloned, this becomes
zephyrproject/modules/audio/sof/zephyrproject which is really not a
`west topdir`. Then west is pointed at:
 real_west_top/modules/audio/sof/zephyrproject/zephyr/samples/subsys/audio/sof
which obviously fails.

This was found while testing
https://thesofproject.github.io/latest/getting_started/build-guide/build-with-zephyr.html
and trying (and failing) to switch away from the older .sh script.

Fix find_west_workspace() to return the output of `west topdir` and use
that to update the `west_top` global.

This bug was introduced from the very start when the script was
converted from shell to python. The shell script did not have this
problem because it could be used without using neither -c nor -p. This
logical difference was mentioned in the code review:
https://github.com/thesofproject/sof/pull/5299#discussion_r801158177

Also remove a couple of accidental creations of a sof/zephyrproject/
directory when sof is already inside a higher zephyrproject.

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2022-06-09 16:53:20 +01:00
mengdonglin 0d5bac87d3 xtensa-build-zephyr.py: avoid showing wrong layout of installed files
If each platform has its own output subdirectory, each platform output
subdirectory will include a key subdirectory with same name. Function
show_installed_files() can fail to find the correct parent for a node
as it only compares the subdir name that can no longer be unique.

This patch uses Anynode to replace Node class to define a new
attribute 'long_name' for a node to store its unique relative path
from the output staging directory (e.g. 'sof/apl/community') and
search for its parent.

The attribute 'name' is still used to store the subdir or file name
for a node (e.g. 'community'), and to display the layout of the
installed firmware files.

Signed-off-by: mengdonglin <mengdong.lin@intel.com>
2022-06-09 14:47:18 +01:00
mengdonglin be007d14ab xtensa-build-zephyr.py: add option to use an output subdirectory for each platform
All firmware files are installed in the same staging directory by default.

This patch adds an option --use-platform-subdir to create an output
subdirectory for each platform.

e.g. $scripts/xtensa-build-zephyr.py apl jsl tgl --use-platform-subdir
The layout of installed files will be like this:

build-sof-staging
├── sof
│   ├── apl
│   │   ├── community
│   │   │   └── sof-apl.ri
│   │   └── sof-apl.ldc
│   ├── tgl
│   │   ├── community
│   │   │   └── sof-tgl.ri
│   │   └── sof-tgl.ldc
│   └── jsl
│       ├── sof-jsl.ldc
│       └── community
│           └── sof-jsl.ri
└── tools
    ├── sof-logger
    └── cavstool.py

Signed-off-by: mengdonglin <mengdong.lin@intel.com>
2022-06-09 14:47:18 +01:00
mengdonglin 85a4c56d92 xtensa-build-zephyr.py: add option to specify key type subdir for output
Add argument option --key-type-subdir <key_type> to choose the output
subdirectory name based on the type of rimage signing key.

User can choose subdirectory name from 'community','dbgkey' and 'none'.
The default one is 'community', the type of default rimage signing key.
We can extend the list later.

When the key type is 'community' or 'dbgkey', the layout of SOF output
directory will be like:
build-sof-staging
├── sof
│   ├── sof-platform1.ldc
│   ├── sof-platform2.ldc
│   ├── ...
│   └── <key_type>
│       ├── sof-platform1.ri
│       ├── sof-platform2.ri
│       └── ...
└── tools
    ├── sof-logger
    └── cavstool.py

When the key type is 'none', there will be no key type subdirectory:
build-sof-staging
├── sof
│   ├── sof-platform1.ldc
│   ├── sof-platform1.ri
│   ├── sof-platform2.ldc
│   ├── sof-platform2.ri
│   └── ...
└── tools
    ├── sof-logger
    └── cavstool.py

Signed-off-by: mengdonglin <mengdong.lin@intel.com>
2022-06-09 14:47:18 +01:00
Marc Herbert 46a5de6f49 xtensa-build-zephyr.py: try to rmdir zephyrproject before cloning
Do not fail when there's only an empty directory left over
because of some early interruption of this script or proxy
misconfiguration, wrong fetch location, etc.

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
2022-06-07 21:39:25 +01:00