Just some cleanups and clarifications on how I'd like to do this going
forward. The main clarifications are around SCHEMA_VERSION, since this
is changing in 0.8 and I want to make sure we're all on the same page.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
We no longer need to filter based on the platform like this. Our tests
know what to do at runtime.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Manifest.from_file() and Manifest.from_file(topdir=<topdir>) produces
different results.
The commit extends the test case to verify that creation of Manifest
object is identical when produced with topdir argument or without, as
long as the topdir provided as argument is identical to what
`util.west_topdir()` would have returned.
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
If Manifest.from_file(topdir=<topdir>) is called with topdir then
Manifest is instantiated with topdir, source_file, and manifest_path.
However, if Manifest.from_file() is called without providing topdir,
then Manifest is instantiated with topdir and source_file only.
This causes the path to manifest repo to become None in later calls and
results in: https://github.com/nrfconnect/sdk-zephyr/pull/341
This is now fixed by ensure `manifest_path` is used for instantiating
Manifest class when using Manifest.from_file().
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
This allows west to be executed as a module with "python3 -m west".
This can be useful if it's desired to execute west under a particular
Python interpreter, e.g. the same one that's used elsewhere in a
larger system.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
While we were working on adding typing annotations, I was unaware that
typing.NamedTuple existed and did not use it. This is a more
convenient alternative to collections.namedtuple since it provides
type checking for fields in the relevant tuples. Switch to it.
Suggested-by: Carles Cufí <carles.cufi@nordicsemi.no>
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Recent versions of tox are complaining with errors that look like
this:
WARNING: test command found but not installed in testenv
cmd: C:\Users\Marti\Envs\west-dev\Scripts\mypy.EXE
env: C:\Users\Marti\src\west\.tox\py3-windows
Maybe you forgot to specify a dependency? See also the allowlist_externals envconfig setting.
DEPRECATION WARNING: this will be an error in tox 4 and above!
Fix that by running all the dependencies through the python binary in
the tox virtualenv. Moving the dependencies out of tox.ini seems to be
necessary too; I couldn't get this to work otherwise. It's more in
keeping with the style in the docs here anyway:
https://tox.readthedocs.io/en/latest/config.html
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Reflect the path-prefix semantics here. This is a bit of an edge case
for the developer but I would argue it's the most intuitive behavior
for the user.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Several users have requested the ability to place a project and its
imports in a designated subdirectory to keep third party code
separated from user-written code. Add this feature using a new key in
an import: map.
Example syntax:
manifest:
projects:
- name: zephyr
url: https://github.com/zephyrproject-rtos/zephyr
import:
path-prefix: zephyr_projects
Semantics:
- the zephyr project, and all imported projects, have zephyr_projects
as an additional path component. e.g. zephyr_projects/zephyr instead
of zephyr as the path for zephyr project
- the path-prefix value cannot escape the workspace, e.g.
path-prefix: ../.. would be an error in the above example.
- the 'path-prefix' value is "cumulative" across imports, e.g.
if the imported manifest in turn does an import with "path-prefix: foo",
the combined directory is "zephyr_projects/foo"
We've already laid most of the groundwork for plumbing this through
the import handling code using the _import_ctx and _import_map
namedtuples. When recursively loading Manifest instances during import
resolution, _import_ctx is a container that allows us to pass
information obtained from "higher" manifests down to "lower" manifests
in the tree. The _import_map namedtuple is just a container for the
import: contents when its value is a dict.
What remains to be done is then to add the current "cumulative" path
prefix to _import_ctx and pass it around to a couple of additional
places where it wasn't previously needed but now is. This requires
generalizing _new_ctx() a bit so we can combine an existing context
with the entire _import_map at a particular "import:" instead of just
using its filter function like we do now.
The one edge case is that since path-prefix affects the "imported"
project itself (zephyr in the above example), we need to make sure to
load it when creating every Project instance. This means we
effectively do it twice, which is a bit wasteful but keeps the code
simple, so it's worthwhile waste.
Fixes: #367
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
It's going to be convenient to call _load_imap() from _load_project()
in order to get the path-prefix out of an "import:" block before we
have a Project instantiated. To do that we will need a way to call it
without the project instance being created.
Make it so with a small refactor.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Make this more readable. Right now the output doesn't include the fact
that the error happened due to a bad import.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Clean up the file based on what zephyr's pylintrc would have
complained about. There's one remaining issue, which is that
ManifestProject doesn't call Project.__init__(), but that should be
fixed with #327 instead.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This is an emacs configuration file that developers can add to
configure their development environments on a project-wide basis.
I would rather avoid leaking one into the main repository, so I'm
going to just ignore it.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Verify the expected behavior related to printing path-related
attributes for the manifest project.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This should come from the manifest.path configuration option, which
may disagree with "self: path:" in the manifest file itself.
Add a --manifest-path-from-yml option as a chicken bit to keep the old
behavior in case anybody is relying on it.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Minor tweaks from testing the 0.7.3 release:
- mention Fedora as a Linux distro as as I've received reports of
active west users on that distro
- update board name for nrf52 board
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Consider a manifest file which uses 'import' on a project, where that
project revision's manifest file contains invalid manifest data.
The manifest-rev branch in that project will be left pointing at the
bogus manifest. The only way out is to update the main west.yml and
point at a new revision, then run west update.
However, main.py refuses to run west update in exactly this situation,
leading to an unusable workspace.
Fix this and add a regression test.
This new ability does open us up to running "west update" even with an
invalid manifest and no imports. We handle this in main.py.
Reported-by: Pete Skeggs <peter.skeggs@nordicsemi.no>
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Experience has borne out the hope that the CI system using GitHub
actions offers the same amount of testing on Linux platforms as
existing shippable infrastructure. It's also configured to run on
macOS and Windows, which is nicer. So there's no need to continue to
run shippable based testing now.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
We've now re-worked the west internals to use pathlib everywhere that
platform-specific path magic was previously relying on canon_path()
or other os.path-based methods to do comparisons or otherwise
canonicalize paths.
We can therefore remove canon_path() and consider the internals of
path handling hopefully satisfactorily reworked. Any further issues
with path handling should be treated as bugs.
Fixes: #273
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Rather than relying on normcase and realpath, just create a resolved
Path object as the module cache key. These have hash keys and
comparison operators which respect platform-specific canonicalization
rules.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Those aren't the same thing on POSIX, and the Project equivalent is
using abspath. Keep it consistent.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Just for completion, this is the last of the non-deprecated public
APIs that accepts a path, so take os.fspath() on it even though nobody
is really ever going to instantiate these but west's CLI itself.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Use pathlib.Path instead. To try to avoid mistakes, type-annotate some of
the auxiliary methods involved in this change to let mypy catch
situations where we're passing a Path to something that expects a str.
Don't go crazy with that, though; this is application code, not API,
so we don't need to get all Java in here.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Modify the APIs so that most any PathLike object can be passed to
public constructors or methods, and the results behave correctly.
Preserve existing Project and Manifest attribute and property
semantics, though: attributes that were str are still str.
We also keep the second argument to ImporterType a str, since
there's no reason not to and it will avoid potentially requiring
future users to call os.fspath() on the argument just to be safe.
This allows us to avoid using west.util.canon_path(), especially to do
path comparisons in Manifest.get_projects(). Instead, let the standard
library do the right thing for us.
Along the way we find another way in which a ManifestProject is not
a Project and should not have been subclassed from it, because we are
forced to ignore type checking for the ManifestProject _path
attribute, since it has the wrong type compared to its parent class.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
These are just for safety, so mypy can yell at us if we break any
rules by assuming an os.PathLike is a str.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Rely on pathlib instead, which handles canonicalization on Windows in
a much better way.
Start using os.fspath(pathlike) instead of str(pathlike). After
studying some CPython sources that deal with path-like objects, this
seems to be the recommended thing to do.
The reason why is that not all os.PathLike objects are going to return
their file system path representations when str() is called on them;
even though the pathlib ones do.
I therefore want to use os.fspath() everywhere instead of str(), even
if it's sometimes safe to use str(). That way, when we inevitably
start copy/pasting code around, it will still do the right thing when
we're dealing with the general case os.PathLike behavior, e.g. from a
user-supplied argument to an API function, which may do something
different in its __str__ than its __fspath__ method.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Move the PathType type alias from west.manifest to west.util and use
it there to type-annotate individual functions as accepting either str
or an os.PathLike.
Change the source code as needed to make this true. Delete a stale
comment mentioning the long-gone bootstrapper while at it.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Add an assertion about a program invariant that will matter when
west.util gets type annotated.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
The subprocess's Popen constructor takes a cwd kwarg. Since v3.6, this
can be a PathLike on POSIX operating systems, but Windows didn't
officially get this feature until 3.7. This ability was back-ported
onto the v3.6 series after v3.6.1-rc1, but v3.6.0 doesn't have it.
To handle that, force cwd to a str on versions below 3.6.1 if it isn't
one already. Add a regression test just for fun, though west's CI
uses 3.6.8, so it passes even without the manifest.py patch.
It seems very unlikely to be an issue in practice since nobody should
be using 3.6.0, but I spent the time tracking this down from the docs
and (irrationally) feel like something ought to be done about it.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Add missing type annotations. Paths are generally 'str' for now,
though there is one place where we can use the PathType value we
defined in a previous commit.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
As we did for previous patches, Project has to be done as a str at
this part of the file, since we haven't gotten around to defining
Project yet.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Now that the internal callers are using the right types,
annotate the validate() and manifest_path() arguments.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This is used internally to pass along an import context in a way that
can't casually be done by a user, since '-' is not a valid character
in an identifier.
Adding asserts about its value won't hurt, and will help convince the
type checker that this is doing the right thing later on.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
The posixpath() property if checks are copy/pasted from abspath, but
they aren't quite right. The _posixpath attribute really cares about
whether self.abspath is None, not self.topdir. Fix it.
This shouldn't affect correctness because self.abspath is non-None
exactly when self.topdir is, but is cleaner readability wise and will
make it easier to type annotate this property.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
These are just duplicating functionality already present in the parent
class. The abspath property has to stay because, unlike Project,
ManifestProject might have a self._path set to None (... which,
looking at it again, is yet another reason it's not sensible to treat
it like a Project).
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Martí Bolívar