We currently don't have a way to group projects with extra semantic
information. There is a use case for this in Zephyr; the ultimate goal
is to allow manifests to not update certain groups of projects by
default, but allow easy update of those groups if the user opts in.
Start to enable this by letting a 'projects' item in a manifest file
have a 'groups' key, which must contain a list of scalar values (str,
int, or float).
For example, in this manifest:
manifest:
# ...
projects:
- name: foo
groups:
- YouAreIt
- name: bar
groups:
- YouAreIt
- MeToo
- name: baz
The projects have these groups lists:
- foo: ['YouAreIt']
- bar: ['YouAreIt', 'MeToo']
- baz: []
As a restriction to give us some room to grow into and generally keep
things sane, 'projects' group values must not contain commas (,),
colons (:), whitespace. Groups cannot start with a '-' either as
that's how we'll be blocking them.
The manifest data are converted to lists of str internally and
associated with the west.manifest.Project objects.
Projects are "active" when any of their groups are not blocked.
Groups are allowed by default, but may be blocked using the manifest,
like this:
manifest:
...
groups: [-thisisblocked,-this-too]
Or via the .west/config workspace local configuration file, like this:
[manifest]
groups = -blocked1,-blocked2,allowed
I.e. via commands like:
$ west config manifest.groups -- -blocked1,-blocked2,allowed
There is also an API routine is_active() for checking if a project is
active, potentially with some extra blocklist and allowlist entries
for groups obtained from elsewhere.
For now, no commands are aware of this. We'll make commands respond
differently to 'inactive' projects in the next patches.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
if running with invalid version (for instance newer than current
west), print (more) user-friendly error such as:
FATAL ERROR: west v1.0 or later is required by the manifest
West version: v0.8.99
Manifest file: west/west.yml
Please upgrade west and retry.
instead of:
File "west/src/west/app/main.py", line 172, in handle_builtin_manifest_load_err
assert args.command not in no_manifest_ok
AssertionError
Signed-off-by: Jacob Siverskog <jacob@teenage.engineering>
this adds support for running python ~/path/to/west/setup.py develop
from other directories.
Signed-off-by: Jacob Siverskog <jacob@teenage.engineering>
Print the git commands on their own lines to make them more visible.
Suggested-by: Joakim Andersson <joakim.andersson@nordicsemi.no>
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Such cases are not well-formed manifests. The value of the self: path:
key must be a nonempty path if present.
Remove the xfail from the relevant test case.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This provides alternatives to the racially laden terms `whitelist`,
used for a list of imports which are included, and `blacklist`, used
for a list of imports which are excluded with `allowlist` and
respectively `blocklist`.
Signed-off-by: Markus Tacker <Markus.Tacker@NordicSemi.no>
Let's use a lexical check instead of escapes_directory(). This
continues to let some users who are symlinking outside the workspace
to keep doing that, even though we don't officially support it in
west.
Also throw in a check for an absolute project path, which should never
happen and is similarly a sign of an attempt to escape the workspace.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Let's make it clearer what happens in this case.
For the sake of not making the command line help too long, let's drop
the point that it can't be combined with -m or --mr and just make it
clear that -l doesn't use MANIFEST_URL. People will find out right
away if they try it that it doesn't work if they combine these
options.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Just some cleanups and clarifications on how I'd like to do this going
forward. The main clarifications are around SCHEMA_VERSION, since this
is changing in 0.8 and I want to make sure we're all on the same page.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
We no longer need to filter based on the platform like this. Our tests
know what to do at runtime.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Manifest.from_file() and Manifest.from_file(topdir=<topdir>) produces
different results.
The commit extends the test case to verify that creation of Manifest
object is identical when produced with topdir argument or without, as
long as the topdir provided as argument is identical to what
`util.west_topdir()` would have returned.
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
If Manifest.from_file(topdir=<topdir>) is called with topdir then
Manifest is instantiated with topdir, source_file, and manifest_path.
However, if Manifest.from_file() is called without providing topdir,
then Manifest is instantiated with topdir and source_file only.
This causes the path to manifest repo to become None in later calls and
results in: https://github.com/nrfconnect/sdk-zephyr/pull/341
This is now fixed by ensure `manifest_path` is used for instantiating
Manifest class when using Manifest.from_file().
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
This allows west to be executed as a module with "python3 -m west".
This can be useful if it's desired to execute west under a particular
Python interpreter, e.g. the same one that's used elsewhere in a
larger system.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
While we were working on adding typing annotations, I was unaware that
typing.NamedTuple existed and did not use it. This is a more
convenient alternative to collections.namedtuple since it provides
type checking for fields in the relevant tuples. Switch to it.
Suggested-by: Carles Cufí <carles.cufi@nordicsemi.no>
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Recent versions of tox are complaining with errors that look like
this:
WARNING: test command found but not installed in testenv
cmd: C:\Users\Marti\Envs\west-dev\Scripts\mypy.EXE
env: C:\Users\Marti\src\west\.tox\py3-windows
Maybe you forgot to specify a dependency? See also the allowlist_externals envconfig setting.
DEPRECATION WARNING: this will be an error in tox 4 and above!
Fix that by running all the dependencies through the python binary in
the tox virtualenv. Moving the dependencies out of tox.ini seems to be
necessary too; I couldn't get this to work otherwise. It's more in
keeping with the style in the docs here anyway:
https://tox.readthedocs.io/en/latest/config.html
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Reflect the path-prefix semantics here. This is a bit of an edge case
for the developer but I would argue it's the most intuitive behavior
for the user.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Several users have requested the ability to place a project and its
imports in a designated subdirectory to keep third party code
separated from user-written code. Add this feature using a new key in
an import: map.
Example syntax:
manifest:
projects:
- name: zephyr
url: https://github.com/zephyrproject-rtos/zephyr
import:
path-prefix: zephyr_projects
Semantics:
- the zephyr project, and all imported projects, have zephyr_projects
as an additional path component. e.g. zephyr_projects/zephyr instead
of zephyr as the path for zephyr project
- the path-prefix value cannot escape the workspace, e.g.
path-prefix: ../.. would be an error in the above example.
- the 'path-prefix' value is "cumulative" across imports, e.g.
if the imported manifest in turn does an import with "path-prefix: foo",
the combined directory is "zephyr_projects/foo"
We've already laid most of the groundwork for plumbing this through
the import handling code using the _import_ctx and _import_map
namedtuples. When recursively loading Manifest instances during import
resolution, _import_ctx is a container that allows us to pass
information obtained from "higher" manifests down to "lower" manifests
in the tree. The _import_map namedtuple is just a container for the
import: contents when its value is a dict.
What remains to be done is then to add the current "cumulative" path
prefix to _import_ctx and pass it around to a couple of additional
places where it wasn't previously needed but now is. This requires
generalizing _new_ctx() a bit so we can combine an existing context
with the entire _import_map at a particular "import:" instead of just
using its filter function like we do now.
The one edge case is that since path-prefix affects the "imported"
project itself (zephyr in the above example), we need to make sure to
load it when creating every Project instance. This means we
effectively do it twice, which is a bit wasteful but keeps the code
simple, so it's worthwhile waste.
Fixes: #367
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
It's going to be convenient to call _load_imap() from _load_project()
in order to get the path-prefix out of an "import:" block before we
have a Project instantiated. To do that we will need a way to call it
without the project instance being created.
Make it so with a small refactor.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Make this more readable. Right now the output doesn't include the fact
that the error happened due to a bad import.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Clean up the file based on what zephyr's pylintrc would have
complained about. There's one remaining issue, which is that
ManifestProject doesn't call Project.__init__(), but that should be
fixed with #327 instead.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This is an emacs configuration file that developers can add to
configure their development environments on a project-wide basis.
I would rather avoid leaking one into the main repository, so I'm
going to just ignore it.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Verify the expected behavior related to printing path-related
attributes for the manifest project.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This should come from the manifest.path configuration option, which
may disagree with "self: path:" in the manifest file itself.
Add a --manifest-path-from-yml option as a chicken bit to keep the old
behavior in case anybody is relying on it.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Minor tweaks from testing the 0.7.3 release:
- mention Fedora as a Linux distro as as I've received reports of
active west users on that distro
- update board name for nrf52 board
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Consider a manifest file which uses 'import' on a project, where that
project revision's manifest file contains invalid manifest data.
The manifest-rev branch in that project will be left pointing at the
bogus manifest. The only way out is to update the main west.yml and
point at a new revision, then run west update.
However, main.py refuses to run west update in exactly this situation,
leading to an unusable workspace.
Fix this and add a regression test.
This new ability does open us up to running "west update" even with an
invalid manifest and no imports. We handle this in main.py.
Reported-by: Pete Skeggs <peter.skeggs@nordicsemi.no>
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Experience has borne out the hope that the CI system using GitHub
actions offers the same amount of testing on Linux platforms as
existing shippable infrastructure. It's also configured to run on
macOS and Windows, which is nicer. So there's no need to continue to
run shippable based testing now.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
We've now re-worked the west internals to use pathlib everywhere that
platform-specific path magic was previously relying on canon_path()
or other os.path-based methods to do comparisons or otherwise
canonicalize paths.
We can therefore remove canon_path() and consider the internals of
path handling hopefully satisfactorily reworked. Any further issues
with path handling should be treated as bugs.
Fixes: #273
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Rather than relying on normcase and realpath, just create a resolved
Path object as the module cache key. These have hash keys and
comparison operators which respect platform-specific canonicalization
rules.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Those aren't the same thing on POSIX, and the Project equivalent is
using abspath. Keep it consistent.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Just for completion, this is the last of the non-deprecated public
APIs that accepts a path, so take os.fspath() on it even though nobody
is really ever going to instantiate these but west's CLI itself.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Use pathlib.Path instead. To try to avoid mistakes, type-annotate some of
the auxiliary methods involved in this change to let mypy catch
situations where we're passing a Path to something that expects a str.
Don't go crazy with that, though; this is application code, not API,
so we don't need to get all Java in here.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Modify the APIs so that most any PathLike object can be passed to
public constructors or methods, and the results behave correctly.
Preserve existing Project and Manifest attribute and property
semantics, though: attributes that were str are still str.
We also keep the second argument to ImporterType a str, since
there's no reason not to and it will avoid potentially requiring
future users to call os.fspath() on the argument just to be safe.
This allows us to avoid using west.util.canon_path(), especially to do
path comparisons in Manifest.get_projects(). Instead, let the standard
library do the right thing for us.
Along the way we find another way in which a ManifestProject is not
a Project and should not have been subclassed from it, because we are
forced to ignore type checking for the ManifestProject _path
attribute, since it has the wrong type compared to its parent class.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
These are just for safety, so mypy can yell at us if we break any
rules by assuming an os.PathLike is a str.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Rely on pathlib instead, which handles canonicalization on Windows in
a much better way.
Start using os.fspath(pathlike) instead of str(pathlike). After
studying some CPython sources that deal with path-like objects, this
seems to be the recommended thing to do.
The reason why is that not all os.PathLike objects are going to return
their file system path representations when str() is called on them;
even though the pathlib ones do.
I therefore want to use os.fspath() everywhere instead of str(), even
if it's sometimes safe to use str(). That way, when we inevitably
start copy/pasting code around, it will still do the right thing when
we're dealing with the general case os.PathLike behavior, e.g. from a
user-supplied argument to an API function, which may do something
different in its __str__ than its __fspath__ method.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Martí Bolívar