On Windows, shlex.split('foo C:\\bar\\baz') results in:
['foo', 'C:barbaz']
This is not what we want; C:\\bar\\baz is a path and backslashes
should be preserved.
This has the result of making this test lead to different results
depending on whether the manifest URL is used by west as it runs or
not.
We could add posix=False to the shlex.split() call on Windows to avoid
this, but it's easier and more readable to just split the arguments
ourselves.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Instead of hard-coding a default revision to fetch from the remote,
get it from the remote URL itself if it's not provided by the user.
If that fails, we ask for the --manifest-rev option to be provided
explicitly.
This makes the code more robust by allowing each remote to dictate
policy instead of making assumptions. It is especially important as
many projects use 'main' instead of 'master' now.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This fix is present in v0.10-branch, but in the wrong commit due to a
botched rebase. Fix it.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
For schema version 0.10:
Allow group-filter values from imported manifests to affect the top
level manifest group filter. Simplify the results so that
Manifest.group_filter is just a list of disabled groups.
Manifests have the same precedence on the final group filter as they
do on project definition: manifests which appear earlier in the import
order have higher precedence on the group filter, just as they have
higher precedence when it comes to whether a project definition comes
from them or is ignored.
Delay loading the manifest.group-filter value from the local
configuration file until we actually need to know its value in
is_active().
Preserve 0.9 semantics if that is explicitly requested, but warn if
there are group filters anywhere.
Add test cases for the updated group-filter behavior, and regression
tests to ensure that 0.9 semantics are preserved when explicitly
requested.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This will be required by a later patch which uses the value '0.9'
explicitly to see if the user is requesting group-filter behavior that
we're going to change. It is also just a good way to tighten up the
error handling.
The intent here is to reject '0.9.99' as an invalid schema version, to
make sure nobody tries to do that. We're bumping the schema from 0.9
to 0.10 as part of the v0.10.0 development branch. Any unreleased
master branches during that time will have to request schema version
0.10 instead of 0.9.99 or something like that.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Move RawGroupType right before its only point of use.
Move GroupFilterType and GroupsType to the top level list of type
aliases, as prep work for using them in more places. Change the '#:'
comment style to '#' while we're here; the '#:' style is reserved for
comments which are also docstrings that end up in Sphinx, which
doesn't apply here.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Add more source code comments describing this internal class's
purpose. Use kwargs for fields whenever initializing an instance.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Change some things related to import map names:
- rename the _import_ctx 'filter_fn' attribute to 'imap_filter',
along with local variables using the same name
- rename _filter_ok() to _imap_filter_allows()
- rename _and_filters() to _compose_imap_filters()
- rename _new_ctx() to _compose_ctx_and_imap()
- adjust the debug logging when an import map filter rejects a project
This helps disambiguate filtering on projects that is due to import
map allowlists and blocklists versus project group filters.
These are all internal names: there are no API changes here.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Freezing, resolving, etc. a manifest file should result in output that
reflects the input's group-filter value too, but it doesn't. Fix that.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
It's annoying to have to specify projects lists in some cases
when all you're trying to do is exercise something else.
Though the number of situations where a manifest with no projects is
useful may be limited, they nonetheless exist, so make it possible.
This is a backwards incompatible change, so bump the schema version.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
API modules like west.manifest are using the standard logging module.
This module prints messages at warning severity or above to stderr by
default.
That prints messages twice when a west command configures its own
logger, like this:
$ west foo
some warning message
WARNING: some warning message
The first line comes from the print to stderr done by the default
handler. The second one is from a ProjectCommandLogHandler.
Disable the first message, which we do not want, by registering a null
handler for the top level. This results in just the
ProjectCommandLogHandler output being shown, like this:
$ west foo
WARNING: some warning message
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Otherwise, setup functions that set git config variables like
user.name are potentially dangerous to run.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
There's just one test that's being skipped this way, and it's not that
slow compared to the rest.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Forcing users to define something unused is not just tedious, it's also
confusing.
Note the name attribute is mostly unused by git submodule too.
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
Add nested _assert() function that raises an exception and reduces
"return False" duplication. Because of the added scaffolding this adds
about 10 lines, on other hand it brings all conditions together in one
place, removes negations and makes it easier to add conditions.
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
This is stale from before the move to group-filter and changing its
semantics to requiring a + and -.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Some test cases are still referring to groups as things that can be
'allowed' and 'blocked'. We switched that language to 'enabled' and
'disabled' in code and documentation. Keep tests up to date.
Do the same in manifest-schema.yml.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
As a result of review of the project groups feature documentation,
change some of the names to be more clear, and be a bit more forgiving
in the error handling.
A project's 'groups' list remains the same. No changes there.
However, we make the following changes:
- 'manifest: groups:' is now 'manifest: group-filter:'
- we add a west.manifest.Manifest.group_filter value, exposing
'manifest: group-filter:' after validation and string conversion
- the config option 'manifest.groups' is now 'manifest.group-filter'
- invalid values in manifest.group-filter are downgraded to warnings
- the west update '--groups' option becomes '--group-filter', with
a slightly shorter name '--gf' if you can't handle that
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Python environments usually do not import importlib.util
unless asked to do so specifically:
>>> import importlib
>>> importlib.util
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: module 'importlib' has no attribute 'util'
Therefore it would be better to import importlib.util explicitly.
Some commands (e.g. west zephyr-export) do not work on my setup
without that change.
Signed-off-by: Ilya Tagunov <tagunil@gmail.com>
During review of the documentation for the 0.9 release, the following
changes to manifest groups were requested:
- groups should be 'enabled' and 'disabled', not 'allowed' or
'blocked'
- enabling a group should be done by prefixing its name with '+'
in "manifest: groups:" or manifest.groups
Projects are still 'active' or 'inactive'.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
If the manifest repository path is placed in a nested subdirectory of
the workspace, shutil.move() does not guarantee success. Strangely,
this works on Linux, even though it probably shouldn't, but it fails
on Windows. Fix that and add a regression test.
Fixes: #463
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This makes 'west update foo' behavior consistent with
'west list foo' in case 'foo' is inactive: we were given a specific
project, so we should "do the thing" even if the project is inactive.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Miscellaneous help text improvements. Add missing information, adjust
misleading bits, and clear up usage.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
We aren't individually tracking the time it takes to update a
project's submodules when we run west update --stats; fix that.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
There's no need for the call_location variable in update_submodules();
Project.git() runs in the project's abspath by default.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Add it to the documented instance attributes within a Project, and
document the NamedTuple itself.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
The _get_submodules() method is setting a 'submodules' attribute on
the Manifest object. It should just return a value instead. The
submodules data is per-project, so there's no need for the Manifest
object to track this, since it already has a list of projects.
Fix that, renaming the method to _load_submodules() as well for
consistency with the other helper procedures called by _load().
Move its definition underneath _validate_project_groups() while we're
here, since that is the previous step inside the caller
(_load_project()).
Align the error handling messages with _load_imap()'s by printing the
name of the project that's causing problems on error.
Convert the docstring to a comment like we do for all the other
private helpers, so west API users calling help() aren't encouraged to
rely on its behavior.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
The type's values are either lists of Submodule objects, or bools.
Wrapping this in Optional means that None is a valid value, which
isn't ideal, as it forces users to check for that type. The value
False is just as good in this case, so let's just use that.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
The Manifest.is_submodule_ok() helper is allowing extra beyond 'name'
and 'path'. Fix that so we make sure that if the dict is present, it
has exactly those keys.
Move it to a top-level helper named _is_submodule_dict_ok(). The '_'
prefix is a hint that it's private; Python methods that don't begin
with '_' are usually considered fair game for public use.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
The submodules were added to the manifest in order to enable
updating projects submodules when performing update operation.
Description of the `west update` command behaviour regarding
submodules:
* Submodules can be defined in the west manifest file
for the specific project, by adding submodules: field.
* Submodules field is not required for every project
and should be added only if it is desired.
* Submodules field may have boolean values (true/false)
or be the list of specific submodules.
* Assigning false value to the submodules field means
that submodules are not used for given project. Example
presented below:
- name: example_project
submodules: false
* Assigning true value to the submodules field means
that all submodules attached by git to the project repository
will be updated. Example presented below:
projects:
- name: example_project
submodules: true
* Assigning list of submodules to the submodules field
means that only those submodules present on the list will be
updated (no matter that there might be any other submodules
attached in git). Every element of the submodules list should
have the name and the path fields (which is an absolute path
to the directory where submodule should be updated). Example
presented below:
- name: example_project
submodules:
- name: first_submodule
path: first_submodule_dir/repo
- name: second_submodule
path: second_submodule_dir/repo
* Submodules for each project are defined independently,
so in some cases they might have assigned true/false values,
lists or not be defined at all.
* When calling `west update` command, submodules for each project
are detected and updated with the `recursive` option.
* By default submodules are updated using `--checkout` strategy,
however it is possible to update them using `--rebase` strategy,
by calling `west update -r` command.
What `west update` command does not do with the submodules:
* Updating submodules does not support `west update -k` option.
In case of using such command, `-k` option is ignored
and submodules are updated using default `--checkout` strategy.
* Assigning true or false value to the submodules field does not
apply to the all projects and their submodules, but only to the all
submodules of a specific project for which submodules field
was defined.
Signed-off-by: Kamil Kasperczyk <kamil.kasperczyk@nordicsemi.no>
West has historically relied on the third-party configobj library for
writing configuration files instead of using the standard library's
configparser module. The reason why is that configobj has round-trip
support for comments.
However, the public reading API uses configparser. Up until now, we
were assuming that the two were compatible for the simple purposes we
needed, and indeed they've proven compatible "enough" that the
different code paths on read vs. write haven't been an issue.
This has become a problem now that we are introducing the
manifest.groups configuration option, though, because its value
contains commas. The configparser and configobj file formats have a
semantic difference between these two options, though:
[section]
foo = "one,two"
bar = one,two
The difference is:
- in configobj, 'foo' is the string "one,two" and 'bar' is the list
['one', 'two']
- in configparser, 'foo' is the string '"one,two"' and bar is the string
'one,two'
Further, the configobj library automatically adds quotes around any
string that contains commas to enforce this distinction.
This is breaking round-trips, since:
west config section.foo one,two # configobj writes "one,two"
west config section.foo # configparser reads '"one,two"'
Looking at it further, configobj development seems to have stalled in
2014, and the most significant user it claims in its
documentation (IPython) has moved on to .py and .json configuration
files.
This isn't worth the hassle. Just drop the configobj dependency and
use configparser everywhere. This will delete comments that users have
added to their configuration files and may otherwise reorder sections,
but having the round-trip semantics correct is more important than
that.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This is currently failing because we are using configparser to read,
but configobj to write. The two don't seem to be compatible.
Setting an option to 'bar,baz' uses configobj, which writes the string
"bar,baz" -- with quotes -- to the config file. Then when we read it
back with configparser, we get the value '"bar,baz"' -- quotes are
not stripped.
This will need to be fixed.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This change is similar to the one already done for west list:
- don't print inactive projects by default
- add an --all argument to include inactive projects
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Add '-f {groups}' and '--all' options. Respect active projects by not
printing inactive ones by default.
This allows printing a project's groups, and hides non-updatable
groups by default.
Here we rely on the fact that groups may not have commas or whitespace
to ensure that the printed-out groups are a single whitespace-free and
comma separated string in the 'west list' output.
For example, in this manifest:
manifest:
# ...
projects:
- name: foo
groups:
- foo-group-1
- foo-group-2
- name: bar
path: path-for-bar
- name: baz
groups:
- baz-group
The output for "west list -f '{name} {groups} {path}'" is the following,
with <SPC> standing for a space character:
foo<SPC>foo-group-1,foo-group-2<SPC>foo
bar<SPC><SPC>path-for-bar
baz<SPC>baz-group<SPC>baz
The empty string in the {groups} field for 'bar' can be picked up by
tools like cut(1) to find the entire list of groups. E.g., this command
$ west list -f '{name} {groups} {path} | cut -d' ' -f 2
will print:
foo-group-1,foo-group-2
baz-group
Additionally, we repurpose the old --all command line option (that used
to print the location of the bootstrapped west repository back in the
0.5.x days) to print projects that would be skipped by 'west update'
by default due to groups or no-groups lists.
We're about to do something similar to several other commands, so
structure the help in the epilog in a way that we'll take advantage of
in subsequent patches.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Add support in "west update" for skipping updates for any project that
the Manifest object says is not active.
In order to allow for overrides, let the user specify additional
groups to add to the allowlist via a new "--group" argument, and
additional groups to add to the blocklist via "--no-group". This is
useful for debugging credential management and doing things like
quickly syncing the entire potential workspace with a command like
"west update --groups all".
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Martí Bolívar