Fixes new `west diff --manifest` option added by commit
0d5ee4eb08 ("app: project: Allow to diff against manifest revision")
Do not pass `project.revision` to `git diff` because `project.revision`
is unresolved user input and does not always resolve to a commit. For
instance, `project.revision` can be the name of a remote branch like
`v3.7-branch` that does not exist locally; then `git diff` fails. Or
even worse: there could be a local branch of the same name which points
to a totally different commit.
This bug was found when discussing issue #747, see 7th comment there for
a longer description of what `manifest-rev` is and how it works.
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
Our release process documentation recommends getting passing tox
results on as many popular linux distributions as time allows. Doing
this by hand is cumbersome, redundant, and error prone.
Add a directory with a helper script that automates the entire process
using docker compose and document its use in MAINTAINERS.rst.
Signed-off-by: Martí Bolívar <mbolivar@amperecomputing.com>
Signed-off-by: Pieter De Gendt <pieter.degendt@basalte.be>
This makes it trivial to inspect, line by line, what got covered.
That is useful when evaluating coverage for a new feature.
Signed-off-by: Martí Bolívar <mbolivar@amperecomputing.com>
Add a testcase to parse a manifest and validate the as_dict output.
Add an invalid manifest file for missing required submodule properties.
Signed-off-by: Pieter De Gendt <pieter.degendt@basalte.be>
It seems natural to recommend "debug" flags when telling the user to
"debug".
More specifically, this makes a big difference in situations like the
one reported in #671
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
Starting with PEP 621, `pyproject.toml` is the standard way of
specifying project metadata.
Also switch to using the version from `pyproject.toml` instead of having
a west.version python module. Adjust documentation for the same.
Update MAINTAINERS.rst to use build for dist building.
Signed-off-by: Ayush Singh <ayush@beagleboard.org>
Fixes 2020 commit a53ec10cf2 ("west diff: only print output for
projects with nonempty diffs")
Fixes#731
At least with git version 2.46.0, --exit-code is 0 when there has been a
merge conflict! git bug or feature? Either way this causes west diff to
wrongly assume there is no diff in that repo.
Two other issues:
1. if returncode is higher than 1 (= a git diff failure) AND -v is used,
then failed.append(project) is never run.
2. stderr is never, ever printed
Try for instance `west diff --fubar` and `west -v --fubar` (which are
possible since commit 73aee32278). One fails and the other does
not. Neither prints any useful error.
The verbosity level should never have any side-effect.
To fix:
- Assume there is a diff when --exit-code is zero AND stdout is not
empty. Treat this the same as --exit-code=1
- Always `failed.append(project)` when --exit-code > 1
- Print stderr when there is a diff or in verbose mode.
This drops one `elif` which also simplifies the logic.
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
Stop using `shlex.split()` in conftest.py as a way to prepare test
commands for subprocess.run() because it does not always work. When we
have quoting issues, just avoid them entirely by passing the test
command as a list.
Replace shlex.split() with a simple str.split() for convenience in
simple cases. This simple split() will immediately fail when trying to
use quote which is working exactly as intended.
This also makes the cmd() interface more similar to subprocess.run() and
its many friends, which is good thing.
This is similar to commit 41007648d9 ("Project.git(list/str): reduce
reliance on shlex.split()") but applied to tests/
Before commit 624880e8ff, shlex.split() was used unconditionally in
conftest.py. As expected, this eventually broke on Windows: shlex is
not compatible across all operating systems and shells.
https://docs.python.org/3/library/subprocess.html#security-considerations
> If the shell is invoked explicitly, via shell=True, it is the
> application’s responsibility to ensure that all whitespace and
> metacharacters are quoted appropriately to avoid shell injection
> vulnerabilities. On _some_ platforms, it is possible to use shlex.quote()
> for this escaping.
(Emphasis mine)
Then commit 624880e8ff made the "interesting" decision to stop using
shlex.split()... only on Windows. This was discussed in
https://github.com/zephyrproject-rtos/west/pull/266#discussion_r284670076
So after this commit, parsing of test commands was delegated to the
shell but... only on Windows! This worked for a long time but eventually
broke testing white spaces for the new `west alias` #716. Rather than
making that Windows-specific hack even more complex with a special case,
finally do the right thing and ask more complex test commands to use a
list. Now we can drop shlex.
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
Commit d842371089 ("app: handle unexpected command name better")
intentionally created a big error handling difference between extensions
and built-in commands. As an unfortunate side effect, it lost the
relevant "manifest import failure" error messages for _extension_
commands. This is especially ironic when the extension command is
missing because the import failed, see example in #726.
Error handling is generally very hard to test comprehensively and even
more so in this complex "bootstrap" area. Rather than trying to refactor
it once again, tweak the existing error messages to gently steer the
user away from (potentially missing) extensions and towards built-in
commands that provide a simpler and better error handling out of the
box: they still show relevant, manifest error messages! Always have.
Also recommand the -vv option which conveniently includes git error
messages like "dubious ownership" since previous commit
23db6e1f3c ("app: re-work logging yet again") from the same giant PR
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
* Add environment variables when running forall for each project.
* Add a "cwd" argument to execute the command in different directory
Signed-off-by: Pieter De Gendt <pieter.degendt@basalte.be>
Add an argument to the west diff command that allows to set the
manifest revision as the merge-base.
Signed-off-by: Pieter De Gendt <pieter.degendt@basalte.be>
Fixes issue reported in PR #710 where most places are hardcoded to
'utf-8' while this one is (Windows) locale-dependent. In the future, we
may want to make this more flexible but the most urgent fix is
consistency: with this commit, manifest decoding should be hardcoded to
'utf-8' everywhere.
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
No functional change: just a lot less duplication & hardcoding.
This can help testing and supporting other encodings. It does not hurt
in any case.
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
Cosmetic fix, no functional change. A little less duplication and
hardcoding.
listdir_at() had an optional "encoding" parameter but it was never used
before now, always defaulting to 'utf-8'.
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
Long overdue; both where still at @v1!
Found by chance in
https://github.com/zephyrproject-rtos/west/actions/runs/9275086435/job/25518959121?pr=711
where all macos-latest configs failed like this:
```
Run actions/setup-python@v1
Error: Version 3.8 with arch x64 not found
Available versions:
(empty)
```
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
There have been many users reports in #558 that `west init` fails on the
Windows NTFS filesystem when some IDEs are running concurrently.
There is no simple and reliable fix for this because:
- The lack of concurrency is a core limitation in the NTFS design that
will not change anytime soon,
- IDEs have no reliable way to know when they should pause scanning the
filesystem.
- Placing temporary files outside the workspace would invite other, complex
and nasty issues with cross-filesystem moves (#558)
Short of a fix, explain why this happens very briefly in `west init -h`
and request users to temporarily close their IDEs. It's the only simple
and universal mitigation available.
Mention the issue number in case someone needs more details, some
evidence or smarter workarounds specific to certain IDEs.
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
Giant commit
https://github.com/zephyrproject-rtos/hal_nxp/commit/f9f0944bc2b4fce
"Update to SDK 2.14" added files with malformed UTF-8, more precisely
with the DEGREE SIGN (°) encoded in 8bit latin1/CP1252. Maybe others.
This crashes `west grep`.
Before this fix:
```
nxp$ west grep 'TEMPERATURE_CONV_FACTOR.*Will give'
Traceback (most recent call last):
File ".local/bin/west", line 33, in <module>
sys.exit(load_entry_point('west', 'console_scripts', 'west')())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "west/src/west/app/main.py", line 1085, in main
app.run(argv or sys.argv[1:])
File "west/src/west/app/main.py", line 244, in run
self.run_command(argv, early_args)
File "west/src/west/app/main.py", line 503, in run_command
self.run_builtin(args, unknown)
File "west/src/west/app/main.py", line 611, in run_builtin
self.cmd.run(args, unknown, self.topdir,
File "west/src/west/commands.py", line 194, in run
self.do_run(args, unknown)
File "west/src/west/app/project.py", line 1765, in do_run
completed_process = self.run_subprocess(
^^^^^^^^^^^^^^^^^^^^
File "west/src/west/commands.py", line 325, in run_subprocess
return subprocess.run(args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/subprocess.py", line 550, in run
stdout, stderr = process.communicate(input, timeout=timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/subprocess.py", line 1209, in communicate
stdout, stderr = self._communicate(input, endtime, timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/subprocess.py", line 2146, in _communicate
stdout = self._translate_newlines(stdout,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/subprocess.py", line 1086, in _translate_newlines
data = data.decode(encoding, errors)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb0 in position 385: invalid start byte
```
After this fix, no crash and no interruption and:
mcux/mcux-sdk/middleware/issdk/sensors/fxpq3115_drv.h:#define FXPQ3115_TEMPERATURE_CONV_FACTOR (256) /* Will give \xb0C */
mcux/mcux-sdk/middleware/issdk/sensors/mpl3115_drv.h:#define MPL3115_TEMPERATURE_CONV_FACTOR (256) /* Will give \xb0C */
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
When interpreted as restructured text, these are causing errors
because the '**' is interpreted as the start of a bold section, which
never ends. This causes build failures in the zephyr docs, which do
treat these as RST.
Wrap them in `` to make the whole thing a literal block to fix this
issue.
Signed-off-by: Martí Bolívar <mbolivar@amperecomputing.com>
Add a new description optional field to the schema. This field is merely
informative, it has no effect whatsoever in the manifest and/or project
processing.
Because descriptions can be multiline strings, additional code has been
added to support prettier dumping of those. PyYAML by default uses
single-line strings in YAML with '\n' characters in it, which look ugly
when dumped. To avoid this, use block-style literals (i.e. the ones
beginning with '|' in the YAML) but only for multi-line description
fields. The rest of the field handling are left untouched to preserve
backwards compatibility.
Note that if we used ruamel we would not need the additional code, since
it supports this natively.
Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
The instructions were out of date, since the version test now uses the
_VALID_SCHEMA_VERS array from manifest.py.
Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
For convenience, Project.git() supports passing either a list (good) or
a string with whitespaces (bad). The latter is parsed with shlex.split()
This saves some typing but the caller has to be extremely careful to
never use the shlex.split() convenience with unsanitized inputs.
Fixes commit 3ac600acaa ("git: clean west ref space after fetching")
where the caller was not careful and concatenated `update-ref -d ` with
unsanitized input, possibly containing special characters as found in
bug #679. Fix this bug by converting the string to a list.
While at it, look for a few other, frequent and risky invocations and
convert their string argument to a list too. The following test hack was
used to semi-automate the search for these other locations:
```
--- a/src/west/manifest.py
+++ b/src/west/manifest.py
@@ -897,6 +897,8 @@ class Project:
:param cwd: directory to run git in (default: ``self.abspath``)
'''
if isinstance(cmd, str):
+ print(cmd)
+ breakpoint()
cmd_list = shlex.split(cmd)
else:
cmd_list = list(cmd)
```
While at it, also convert to a list a couple non-risky but very frequent
invocations. This speeds up the test hack above.
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
It's not unusual to make "quick and dirty", temporary changes in a git
repo that is on the manifest: either add some untracked files or create
a branch. As expected, this makes the repository show in `west
compare`. If such repos and the manifest repo are the only repos
appearing in the `west compare` output, then no manifest-rev + HEAD
banner ever appeared. When no banner ever appears it's really not
obvious which repos are on the manifest-rev vs not. In other words: the
main `west compare` feature is defeated.
Clear this doubt by always showing the banner, even when `HEAD` and
`manifest-rev` are the same. See sample output below.
I think the original design idea was to follow diff's "spirit" not to
show anything that's identical and to save as many lines as
possible. However I don't think it works in this particular case because
invoking `git status` for a repo _without_ showing where its HEAD is at
is way too subtle, _especially_ when there is no other repo with a
banner to provide a comparison point and some contrast. Explicitly and
consistently prefixing every `git status` output with a manifest-rev
banner is much more clear and obvious.
Moreover, `git status` output is relatively verbose already so always
prefixing it with a 2 lines long banner makes negligible difference to
the total.
Before:
```
$ west compare
=== hal_xtensa (modules/hal/xtensa):
--- status:
On branch somebranch
nothing to commit, working tree clean
```
After:
```
$ west compare
=== hal_xtensa (modules/hal/xtensa):
--- manifest-rev: 41a631d4aeee (somebranch, upstream/master) cmake: enable using SOC_TOOLCHAIN_NAME ...
HEAD: 41a631d4aeee (somebranch, upstream/master) cmake: enable using SOC_TOOLCHAIN_NAME ...
--- status:
On branch somebranch
nothing to commit, working tree clean
```
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
The tox indirection layer is convenient except for the usual problems
caused by too many layers of indirection:
- Add some examples and show how it is still possible to use important
pytest features. This is also useful to boost people familiar with
Python but not with pytest.
- To stop developers wasting their time trying, document a major
limitation with the current approach: impossible to run the west
code in a debugger when it's started from a test.
- Show how the "printf" alternative can work.
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
This is required to test code that includes temporary test hacks that
don't pass style checks.
People write ugly code first. Then they make it prettier.
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
When setting up a project with west, the target directory may not be
initialized correctly. In the typical case, if a directory named
`./zephyr/` already exists, the user may find that checkout files are
located at `./zephyr/manifest-tmp/*` instead of the expected
`./zephyr/*`.
This patch will abort and refuse to complete `west init` if the
destination directory alread exists. This check would ideally occur
before the potentially lengthy clone operation, but `manifest_path` is
derived from the files retrieved...
NOTE: If the project quotes a value other than `zephyr` for
`manifest.self.path` in `/west.yml`, then this will affect that
directory instead.
Steps to reproduce before this patch:
mkdir ./zephyr/
west init ./ -m https://github.com/zephyrproject-rtos/zephyr.git
ls -l ./zephyr/
Possible fix for some of the symptoms described in #558
Signed-off-by: Attie Grande <attie.grande@argentum-systems.co.uk>
Fixes commit 1d220bbd62 ("commands: add grep")
Fix misplaced closing parenthese in:
```
shlex.split(self.config.get(f'grep.{tool}-args'), '')
```
The empty string is meant as a default value for config.get() but it was
passed as a second argument to shlex.split() by mistake.
Funny enough this hangs forever:
```
>>> shlex.split(None, comments='')
<stdin>:1: DeprecationWarning: Passing None for 's' to shlex.split() is deprecated.
(hangs forever)
```
PS: debuggers rulez
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
Marc Herbert