Add nested _assert() function that raises an exception and reduces
"return False" duplication. Because of the added scaffolding this adds
about 10 lines, on other hand it brings all conditions together in one
place, removes negations and makes it easier to add conditions.
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
This is stale from before the move to group-filter and changing its
semantics to requiring a + and -.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Some test cases are still referring to groups as things that can be
'allowed' and 'blocked'. We switched that language to 'enabled' and
'disabled' in code and documentation. Keep tests up to date.
Do the same in manifest-schema.yml.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
As a result of review of the project groups feature documentation,
change some of the names to be more clear, and be a bit more forgiving
in the error handling.
A project's 'groups' list remains the same. No changes there.
However, we make the following changes:
- 'manifest: groups:' is now 'manifest: group-filter:'
- we add a west.manifest.Manifest.group_filter value, exposing
'manifest: group-filter:' after validation and string conversion
- the config option 'manifest.groups' is now 'manifest.group-filter'
- invalid values in manifest.group-filter are downgraded to warnings
- the west update '--groups' option becomes '--group-filter', with
a slightly shorter name '--gf' if you can't handle that
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Python environments usually do not import importlib.util
unless asked to do so specifically:
>>> import importlib
>>> importlib.util
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: module 'importlib' has no attribute 'util'
Therefore it would be better to import importlib.util explicitly.
Some commands (e.g. west zephyr-export) do not work on my setup
without that change.
Signed-off-by: Ilya Tagunov <tagunil@gmail.com>
During review of the documentation for the 0.9 release, the following
changes to manifest groups were requested:
- groups should be 'enabled' and 'disabled', not 'allowed' or
'blocked'
- enabling a group should be done by prefixing its name with '+'
in "manifest: groups:" or manifest.groups
Projects are still 'active' or 'inactive'.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
If the manifest repository path is placed in a nested subdirectory of
the workspace, shutil.move() does not guarantee success. Strangely,
this works on Linux, even though it probably shouldn't, but it fails
on Windows. Fix that and add a regression test.
Fixes: #463
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This makes 'west update foo' behavior consistent with
'west list foo' in case 'foo' is inactive: we were given a specific
project, so we should "do the thing" even if the project is inactive.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Miscellaneous help text improvements. Add missing information, adjust
misleading bits, and clear up usage.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
We aren't individually tracking the time it takes to update a
project's submodules when we run west update --stats; fix that.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
There's no need for the call_location variable in update_submodules();
Project.git() runs in the project's abspath by default.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Add it to the documented instance attributes within a Project, and
document the NamedTuple itself.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
The _get_submodules() method is setting a 'submodules' attribute on
the Manifest object. It should just return a value instead. The
submodules data is per-project, so there's no need for the Manifest
object to track this, since it already has a list of projects.
Fix that, renaming the method to _load_submodules() as well for
consistency with the other helper procedures called by _load().
Move its definition underneath _validate_project_groups() while we're
here, since that is the previous step inside the caller
(_load_project()).
Align the error handling messages with _load_imap()'s by printing the
name of the project that's causing problems on error.
Convert the docstring to a comment like we do for all the other
private helpers, so west API users calling help() aren't encouraged to
rely on its behavior.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
The type's values are either lists of Submodule objects, or bools.
Wrapping this in Optional means that None is a valid value, which
isn't ideal, as it forces users to check for that type. The value
False is just as good in this case, so let's just use that.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
The Manifest.is_submodule_ok() helper is allowing extra beyond 'name'
and 'path'. Fix that so we make sure that if the dict is present, it
has exactly those keys.
Move it to a top-level helper named _is_submodule_dict_ok(). The '_'
prefix is a hint that it's private; Python methods that don't begin
with '_' are usually considered fair game for public use.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
The submodules were added to the manifest in order to enable
updating projects submodules when performing update operation.
Description of the `west update` command behaviour regarding
submodules:
* Submodules can be defined in the west manifest file
for the specific project, by adding submodules: field.
* Submodules field is not required for every project
and should be added only if it is desired.
* Submodules field may have boolean values (true/false)
or be the list of specific submodules.
* Assigning false value to the submodules field means
that submodules are not used for given project. Example
presented below:
- name: example_project
submodules: false
* Assigning true value to the submodules field means
that all submodules attached by git to the project repository
will be updated. Example presented below:
projects:
- name: example_project
submodules: true
* Assigning list of submodules to the submodules field
means that only those submodules present on the list will be
updated (no matter that there might be any other submodules
attached in git). Every element of the submodules list should
have the name and the path fields (which is an absolute path
to the directory where submodule should be updated). Example
presented below:
- name: example_project
submodules:
- name: first_submodule
path: first_submodule_dir/repo
- name: second_submodule
path: second_submodule_dir/repo
* Submodules for each project are defined independently,
so in some cases they might have assigned true/false values,
lists or not be defined at all.
* When calling `west update` command, submodules for each project
are detected and updated with the `recursive` option.
* By default submodules are updated using `--checkout` strategy,
however it is possible to update them using `--rebase` strategy,
by calling `west update -r` command.
What `west update` command does not do with the submodules:
* Updating submodules does not support `west update -k` option.
In case of using such command, `-k` option is ignored
and submodules are updated using default `--checkout` strategy.
* Assigning true or false value to the submodules field does not
apply to the all projects and their submodules, but only to the all
submodules of a specific project for which submodules field
was defined.
Signed-off-by: Kamil Kasperczyk <kamil.kasperczyk@nordicsemi.no>
West has historically relied on the third-party configobj library for
writing configuration files instead of using the standard library's
configparser module. The reason why is that configobj has round-trip
support for comments.
However, the public reading API uses configparser. Up until now, we
were assuming that the two were compatible for the simple purposes we
needed, and indeed they've proven compatible "enough" that the
different code paths on read vs. write haven't been an issue.
This has become a problem now that we are introducing the
manifest.groups configuration option, though, because its value
contains commas. The configparser and configobj file formats have a
semantic difference between these two options, though:
[section]
foo = "one,two"
bar = one,two
The difference is:
- in configobj, 'foo' is the string "one,two" and 'bar' is the list
['one', 'two']
- in configparser, 'foo' is the string '"one,two"' and bar is the string
'one,two'
Further, the configobj library automatically adds quotes around any
string that contains commas to enforce this distinction.
This is breaking round-trips, since:
west config section.foo one,two # configobj writes "one,two"
west config section.foo # configparser reads '"one,two"'
Looking at it further, configobj development seems to have stalled in
2014, and the most significant user it claims in its
documentation (IPython) has moved on to .py and .json configuration
files.
This isn't worth the hassle. Just drop the configobj dependency and
use configparser everywhere. This will delete comments that users have
added to their configuration files and may otherwise reorder sections,
but having the round-trip semantics correct is more important than
that.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This is currently failing because we are using configparser to read,
but configobj to write. The two don't seem to be compatible.
Setting an option to 'bar,baz' uses configobj, which writes the string
"bar,baz" -- with quotes -- to the config file. Then when we read it
back with configparser, we get the value '"bar,baz"' -- quotes are
not stripped.
This will need to be fixed.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This change is similar to the one already done for west list:
- don't print inactive projects by default
- add an --all argument to include inactive projects
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Add '-f {groups}' and '--all' options. Respect active projects by not
printing inactive ones by default.
This allows printing a project's groups, and hides non-updatable
groups by default.
Here we rely on the fact that groups may not have commas or whitespace
to ensure that the printed-out groups are a single whitespace-free and
comma separated string in the 'west list' output.
For example, in this manifest:
manifest:
# ...
projects:
- name: foo
groups:
- foo-group-1
- foo-group-2
- name: bar
path: path-for-bar
- name: baz
groups:
- baz-group
The output for "west list -f '{name} {groups} {path}'" is the following,
with <SPC> standing for a space character:
foo<SPC>foo-group-1,foo-group-2<SPC>foo
bar<SPC><SPC>path-for-bar
baz<SPC>baz-group<SPC>baz
The empty string in the {groups} field for 'bar' can be picked up by
tools like cut(1) to find the entire list of groups. E.g., this command
$ west list -f '{name} {groups} {path} | cut -d' ' -f 2
will print:
foo-group-1,foo-group-2
baz-group
Additionally, we repurpose the old --all command line option (that used
to print the location of the bootstrapped west repository back in the
0.5.x days) to print projects that would be skipped by 'west update'
by default due to groups or no-groups lists.
We're about to do something similar to several other commands, so
structure the help in the epilog in a way that we'll take advantage of
in subsequent patches.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Add support in "west update" for skipping updates for any project that
the Manifest object says is not active.
In order to allow for overrides, let the user specify additional
groups to add to the allowlist via a new "--group" argument, and
additional groups to add to the blocklist via "--no-group". This is
useful for debugging credential management and doing things like
quickly syncing the entire potential workspace with a command like
"west update --groups all".
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
We currently don't have a way to group projects with extra semantic
information. There is a use case for this in Zephyr; the ultimate goal
is to allow manifests to not update certain groups of projects by
default, but allow easy update of those groups if the user opts in.
Start to enable this by letting a 'projects' item in a manifest file
have a 'groups' key, which must contain a list of scalar values (str,
int, or float).
For example, in this manifest:
manifest:
# ...
projects:
- name: foo
groups:
- YouAreIt
- name: bar
groups:
- YouAreIt
- MeToo
- name: baz
The projects have these groups lists:
- foo: ['YouAreIt']
- bar: ['YouAreIt', 'MeToo']
- baz: []
As a restriction to give us some room to grow into and generally keep
things sane, 'projects' group values must not contain commas (,),
colons (:), whitespace. Groups cannot start with a '-' either as
that's how we'll be blocking them.
The manifest data are converted to lists of str internally and
associated with the west.manifest.Project objects.
Projects are "active" when any of their groups are not blocked.
Groups are allowed by default, but may be blocked using the manifest,
like this:
manifest:
...
groups: [-thisisblocked,-this-too]
Or via the .west/config workspace local configuration file, like this:
[manifest]
groups = -blocked1,-blocked2,allowed
I.e. via commands like:
$ west config manifest.groups -- -blocked1,-blocked2,allowed
There is also an API routine is_active() for checking if a project is
active, potentially with some extra blocklist and allowlist entries
for groups obtained from elsewhere.
For now, no commands are aware of this. We'll make commands respond
differently to 'inactive' projects in the next patches.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
if running with invalid version (for instance newer than current
west), print (more) user-friendly error such as:
FATAL ERROR: west v1.0 or later is required by the manifest
West version: v0.8.99
Manifest file: west/west.yml
Please upgrade west and retry.
instead of:
File "west/src/west/app/main.py", line 172, in handle_builtin_manifest_load_err
assert args.command not in no_manifest_ok
AssertionError
Signed-off-by: Jacob Siverskog <jacob@teenage.engineering>
this adds support for running python ~/path/to/west/setup.py develop
from other directories.
Signed-off-by: Jacob Siverskog <jacob@teenage.engineering>
Print the git commands on their own lines to make them more visible.
Suggested-by: Joakim Andersson <joakim.andersson@nordicsemi.no>
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Such cases are not well-formed manifests. The value of the self: path:
key must be a nonempty path if present.
Remove the xfail from the relevant test case.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This provides alternatives to the racially laden terms `whitelist`,
used for a list of imports which are included, and `blacklist`, used
for a list of imports which are excluded with `allowlist` and
respectively `blocklist`.
Signed-off-by: Markus Tacker <Markus.Tacker@NordicSemi.no>
Let's use a lexical check instead of escapes_directory(). This
continues to let some users who are symlinking outside the workspace
to keep doing that, even though we don't officially support it in
west.
Also throw in a check for an absolute project path, which should never
happen and is similarly a sign of an attempt to escape the workspace.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Let's make it clearer what happens in this case.
For the sake of not making the command line help too long, let's drop
the point that it can't be combined with -m or --mr and just make it
clear that -l doesn't use MANIFEST_URL. People will find out right
away if they try it that it doesn't work if they combine these
options.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Just some cleanups and clarifications on how I'd like to do this going
forward. The main clarifications are around SCHEMA_VERSION, since this
is changing in 0.8 and I want to make sure we're all on the same page.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
We no longer need to filter based on the platform like this. Our tests
know what to do at runtime.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Manifest.from_file() and Manifest.from_file(topdir=<topdir>) produces
different results.
The commit extends the test case to verify that creation of Manifest
object is identical when produced with topdir argument or without, as
long as the topdir provided as argument is identical to what
`util.west_topdir()` would have returned.
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
If Manifest.from_file(topdir=<topdir>) is called with topdir then
Manifest is instantiated with topdir, source_file, and manifest_path.
However, if Manifest.from_file() is called without providing topdir,
then Manifest is instantiated with topdir and source_file only.
This causes the path to manifest repo to become None in later calls and
results in: https://github.com/nrfconnect/sdk-zephyr/pull/341
This is now fixed by ensure `manifest_path` is used for instantiating
Manifest class when using Manifest.from_file().
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
This allows west to be executed as a module with "python3 -m west".
This can be useful if it's desired to execute west under a particular
Python interpreter, e.g. the same one that's used elsewhere in a
larger system.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
West Test