Commit Graph

801 Commits

Author SHA1 Message Date
Michael Zimmermann 80bf5cf381 add support for manifest filenames other than west.yml
Signed-off-by: Michael Zimmermann <michael.zimmermann@grandcentrix.net>
2020-08-26 16:44:59 -07:00
Torsten Rasmussen 6350ca4f86 tests: extended test case for Manifest.from_file()
Manifest.from_file() and Manifest.from_file(topdir=<topdir>) produces
different results.

The commit extends the test case to verify that creation of Manifest
object is identical when produced with topdir argument or without, as
long as the topdir provided as argument is identical to what
`util.west_topdir()` would have returned.

Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
2020-08-19 15:03:28 -07:00
Torsten Rasmussen 60d9c4a399 manifest: adding manifest_path to kwargs when calling Manifest.__init__
If Manifest.from_file(topdir=<topdir>) is called with topdir then
Manifest is instantiated with topdir, source_file, and manifest_path.

However, if Manifest.from_file() is called without providing topdir,
then Manifest is instantiated with topdir and source_file only.

This causes the path to manifest repo to become None in later calls and
results in: https://github.com/nrfconnect/sdk-zephyr/pull/341

This is now fixed by ensure `manifest_path` is used for instantiating
Manifest class when using Manifest.from_file().

Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
2020-08-19 15:03:28 -07:00
Martí Bolívar 05106d4f9b add __main__.py
This allows west to be executed as a module with "python3 -m west".

This can be useful if it's desired to execute west under a particular
Python interpreter, e.g. the same one that's used elsewhere in a
larger system.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2020-08-19 08:59:43 -07:00
Martí Bolívar a8a706e69f manifest: use typing.NamedTuple
While we were working on adding typing annotations, I was unaware that
typing.NamedTuple existed and did not use it.  This is a more
convenient alternative to collections.namedtuple since it provides
type checking for fields in the relevant tuples. Switch to it.

Suggested-by: Carles Cufí <carles.cufi@nordicsemi.no>
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2020-08-17 12:07:47 -07:00
Martí Bolívar 64daba327f tests: clean up dependency handling
Recent versions of tox are complaining with errors that look like
this:

  WARNING: test command found but not installed in testenv
    cmd: C:\Users\Marti\Envs\west-dev\Scripts\mypy.EXE
    env: C:\Users\Marti\src\west\.tox\py3-windows
  Maybe you forgot to specify a dependency? See also the allowlist_externals envconfig setting.

  DEPRECATION WARNING: this will be an error in tox 4 and above!

Fix that by running all the dependencies through the python binary in
the tox virtualenv. Moving the dependencies out of tox.ini seems to be
necessary too; I couldn't get this to work otherwise. It's more in
keeping with the style in the docs here anyway:

https://tox.readthedocs.io/en/latest/config.html

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2020-08-14 14:44:52 -07:00
Martí Bolívar 925eb85d3b manifest: update ImportFlag docstring
Reflect the path-prefix semantics here. This is a bit of an edge case
for the developer but I would argue it's the most intuitive behavior
for the user.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2020-08-14 14:44:52 -07:00
Martí Bolívar c0af04fcec tests: add test cases for import: path-prefix
Exercise the semantics mentioned in the commit log for the previous
patch.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2020-08-14 14:44:52 -07:00
Martí Bolívar 3171afd0e6 manifest: add 'path-prefix' key to import: map
Several users have requested the ability to place a project and its
imports in a designated subdirectory to keep third party code
separated from user-written code. Add this feature using a new key in
an import: map.

Example syntax:

  manifest:
    projects:
    - name: zephyr
      url: https://github.com/zephyrproject-rtos/zephyr
      import:
        path-prefix: zephyr_projects

Semantics:

 - the zephyr project, and all imported projects, have zephyr_projects
   as an additional path component. e.g. zephyr_projects/zephyr instead
   of zephyr as the path for zephyr project

 - the path-prefix value cannot escape the workspace, e.g.
   path-prefix: ../.. would be an error in the above example.

 - the 'path-prefix' value is "cumulative" across imports, e.g.
   if the imported manifest in turn does an import with "path-prefix: foo",
   the combined directory is "zephyr_projects/foo"

We've already laid most of the groundwork for plumbing this through
the import handling code using the _import_ctx and _import_map
namedtuples. When recursively loading Manifest instances during import
resolution, _import_ctx is a container that allows us to pass
information obtained from "higher" manifests down to "lower" manifests
in the tree. The _import_map namedtuple is just a container for the
import: contents when its value is a dict.

What remains to be done is then to add the current "cumulative" path
prefix to _import_ctx and pass it around to a couple of additional
places where it wasn't previously needed but now is. This requires
generalizing _new_ctx() a bit so we can combine an existing context
with the entire _import_map at a particular "import:" instead of just
using its filter function like we do now.

The one edge case is that since path-prefix affects the "imported"
project itself (zephyr in the above example), we need to make sure to
load it when creating every Project instance. This means we
effectively do it twice, which is a bit wasteful but keeps the code
simple, so it's worthwhile waste.

Fixes: #367
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2020-08-14 14:44:52 -07:00
Martí Bolívar 25078d9eb8 manifest: adjust some implementation details
It's going to be convenient to call _load_imap() from _load_project()
in order to get the path-prefix out of an "import:" block before we
have a Project instantiated. To do that we will need a way to call it
without the project instance being created.

Make it so with a small refactor.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2020-08-14 14:44:52 -07:00
Martí Bolívar 66efd497cf ManifestCommand: tweak manifest import error printline
Make this more readable. Right now the output doesn't include the fact
that the error happened due to a bad import.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2020-08-14 14:44:52 -07:00
Martí Bolívar 309b134009 manifest: debug log project paths
This will be useful when supporting users making use of path-prefix.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2020-08-14 14:44:52 -07:00
Martí Bolívar 8cc2127a41 manifest: clean up some zephyr pylintrc complaints
Clean up the file based on what zephyr's pylintrc would have
complained about. There's one remaining issue, which is that
ManifestProject doesn't call Project.__init__(), but that should be
fixed with #327 instead.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2020-08-14 14:44:52 -07:00
Martí Bolívar f386606a1c manifest: add missing return type
The _new_ctx() return type annotation was incorrectly not added.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2020-08-14 14:44:52 -07:00
Martí Bolívar 91fada5e23 tests: minor cleanups
Fix rev_parse() whitespace and make a misleading comment more clear.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2020-08-14 14:44:52 -07:00
Martí Bolívar f393a18e1c .gitignore: add .dir-locals.el
This is an emacs configuration file that developers can add to
configure their development environments on a project-wide basis.

I would rather avoid leaking one into the main repository, so I'm
going to just ignore it.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2020-08-14 14:44:52 -07:00
Martí Bolívar 27c2dd02c9 tests: add regression tests for 'west list manifest'
Verify the expected behavior related to printing path-related
attributes for the manifest project.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2020-08-10 07:58:45 -07:00
Martí Bolívar 0bcfad5ce2 west list: fix manifest path
This should come from the manifest.path configuration option, which
may disagree with "self: path:" in the manifest file itself.

Add a --manifest-path-from-yml option as a chicken bit to keep the old
behavior in case anybody is relying on it.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2020-08-10 07:58:45 -07:00
Martí Bolívar 97df7aa7ce MAINTAINERS.rst: updates following 0.7.3
Minor tweaks from testing the 0.7.3 release:

- mention Fedora as a Linux distro as as I've received reports of
  active west users on that distro

- update board name for nrf52 board

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2020-08-05 16:21:42 +02:00
Martí Bolívar d8e6065067 app: fix import error followed by update
Consider a manifest file which uses 'import' on a project, where that
project revision's manifest file contains invalid manifest data.
The manifest-rev branch in that project will be left pointing at the
bogus manifest. The only way out is to update the main west.yml and
point at a new revision, then run west update.

However, main.py refuses to run west update in exactly this situation,
leading to an unusable workspace.

Fix this and add a regression test.

This new ability does open us up to running "west update" even with an
invalid manifest and no imports. We handle this in main.py.

Reported-by: Pete Skeggs <peter.skeggs@nordicsemi.no>
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2020-07-14 23:00:33 +02:00
Martí Bolívar 6524bffac8 conftest: make it more os.PathLike friendly
Sprinkle around some more os.fspath() calls.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2020-07-14 23:00:33 +02:00
Martí Bolívar 24ab747ac3 remove .shippable.yml
Experience has borne out the hope that the CI system using GitHub
actions offers the same amount of testing on Linux platforms as
existing shippable infrastructure. It's also configured to run on
macOS and Windows, which is nicer. So there's no need to continue to
run shippable based testing now.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2020-07-08 15:41:56 -07:00
Martí Bolívar f466818a95 util: remove canon_path()
We've now re-worked the west internals to use pathlib everywhere that
platform-specific path magic was previously relying on canon_path()
or other os.path-based methods to do comparisons or otherwise
canonicalize paths.

We can therefore remove canon_path() and consider the internals of
path handling hopefully satisfactorily reworked. Any further issues
with path handling should be treated as bugs.

Fixes: #273
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2020-07-08 14:00:52 -07:00
Martí Bolívar ef6cc1d3c4 commands: modify extension command internals
Rather than relying on normcase and realpath, just create a resolved
Path object as the module cache key. These have hash keys and
comparison operators which respect platform-specific canonicalization
rules.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2020-07-08 14:00:52 -07:00
Martí Bolívar 4e85b171a0 manifest: replace a realpath with an abspath in ManifestProject
Those aren't the same thing on POSIX, and the Project equivalent is
using abspath. Keep it consistent.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2020-07-08 14:00:52 -07:00
Martí Bolívar 476f4fca15 commands: accept PathLike topdir
Just for completion, this is the last of the non-deprecated public
APIs that accepts a path, so take os.fspath() on it even though nobody
is really ever going to instantiate these but west's CLI itself.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2020-07-08 14:00:52 -07:00
Martí Bolívar f5bb5b2997 project.py: stop using canon_path()
Use pathlib.Path instead. To try to avoid mistakes, type-annotate some of
the auxiliary methods involved in this change to let mypy catch
situations where we're passing a Path to something that expects a str.
Don't go crazy with that, though; this is application code, not API,
so we don't need to get all Java in here.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2020-07-08 14:00:52 -07:00
Martí Bolívar 90b506974c app/project: add missing f for f-string
Whoops.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2020-07-08 14:00:52 -07:00
Martí Bolívar 46f583c662 manifest: accept os.PathLike, produce str
Modify the APIs so that most any PathLike object can be passed to
public constructors or methods, and the results behave correctly.

Preserve existing Project and Manifest attribute and property
semantics, though: attributes that were str are still str.

We also keep the second argument to ImporterType a str, since
there's no reason not to and it will avoid potentially requiring
future users to call os.fspath() on the argument just to be safe.

This allows us to avoid using west.util.canon_path(), especially to do
path comparisons in Manifest.get_projects(). Instead, let the standard
library do the right thing for us.

Along the way we find another way in which a ManifestProject is not
a Project and should not have been subclassed from it, because we are
forced to ignore type checking for the ManifestProject _path
attribute, since it has the wrong type compared to its parent class.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2020-07-08 14:00:52 -07:00
Martí Bolívar 53c4ad6af7 configuration: add type annotations
These are just for safety, so mypy can yell at us if we break any
rules by assuming an os.PathLike is a str.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2020-07-08 14:00:52 -07:00
Martí Bolívar ff314ad342 configuration: don't use canon_path()
Rely on pathlib instead, which handles canonicalization on Windows in
a much better way.

Start using os.fspath(pathlike) instead of str(pathlike). After
studying some CPython sources that deal with path-like objects, this
seems to be the recommended thing to do.

The reason why is that not all os.PathLike objects are going to return
their file system path representations when str() is called on them;
even though the pathlib ones do.

I therefore want to use os.fspath() everywhere instead of str(), even
if it's sometimes safe to use str(). That way, when we inevitably
start copy/pasting code around, it will still do the right thing when
we're dealing with the general case os.PathLike behavior, e.g. from a
user-supplied argument to an API function, which may do something
different in its __str__ than its __fspath__ method.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2020-07-08 14:00:52 -07:00
Martí Bolívar fe50aff3a3 configuration: remove deprecated read_config kwarg
The config_file kwarg has been deprecated for a while; delete it.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2020-07-08 14:00:52 -07:00
Martí Bolívar 3580dd2555 test_config: don't use obsolete kwarg
Use configfile, not config_file.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2020-07-08 14:00:52 -07:00
Martí Bolívar f6331aba4f util: accept PathLike where relevant
Move the PathType type alias from west.manifest to west.util and use
it there to type-annotate individual functions as accepting either str
or an os.PathLike.

Change the source code as needed to make this true. Delete a stale
comment mentioning the long-gone bootstrapper while at it.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2020-07-08 14:00:52 -07:00
Martí Bolívar e637e2ac83 manifest: use a crystal ball to aid the next commit
Add an assertion about a program invariant that will matter when
west.util gets type annotated.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2020-07-08 14:00:52 -07:00
Martí Bolívar a51b7774e4 manifest: fix Project.git() cwd on windows / 3.6.0
The subprocess's Popen constructor takes a cwd kwarg. Since v3.6, this
can be a PathLike on POSIX operating systems, but Windows didn't
officially get this feature until 3.7. This ability was back-ported
onto the v3.6 series after v3.6.1-rc1, but v3.6.0 doesn't have it.

To handle that, force cwd to a str on versions below 3.6.1 if it isn't
one already. Add a regression test just for fun, though west's CI
uses 3.6.8, so it passes even without the manifest.py patch.

It seems very unlikely to be an issue in practice since nobody should
be using 3.6.0, but I spent the time tracking this down from the docs
and (irrationally) feel like something ought to be done about it.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2020-07-08 14:00:52 -07:00
Martí Bolívar 25d24287a8 .gitignore: add htmlcov/
This includes coverage data in human-readable format. It's generated
by tox.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2020-06-30 12:04:35 -07:00
Martí Bolívar 2c0567cb83 manifest: clarify validate() docstring
This is unclear; try to fix it.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2020-06-24 09:59:27 -07:00
Martí Bolívar e0eba2b7dc manifest: type annotate Manifest
This completes the initial type annotation of this module.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2020-06-24 09:59:27 -07:00
Martí Bolívar 9492bb62c3 manifest: type annotate Project and ManifestProject
Add missing type annotations. Paths are generally 'str' for now,
though there is one place where we can use the PathType value we
defined in a previous commit.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2020-06-24 09:59:27 -07:00
Martí Bolívar 0020d8211e manifest: type annotate exception types
As we did for previous patches, Project has to be done as a str at
this part of the file, since we haven't gotten around to defining
Project yet.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2020-06-24 09:59:27 -07:00
Martí Bolívar 951317b4ae manifest: type annotate public functions
Now that the internal callers are using the right types,
annotate the validate() and manifest_path() arguments.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2020-06-24 09:59:27 -07:00
Martí Bolívar 84b810c7ff manifest: type annotate internal functions
This is step 1 to annotating the public APIs.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2020-06-24 09:59:27 -07:00
Martí Bolívar 54e4611389 manifest: type check 'import-context' kwarg
This is used internally to pass along an import context in a way that
can't casually be done by a user, since '-' is not a valid character
in an identifier.

Adding asserts about its value won't hurt, and will help convince the
type checker that this is doing the right thing later on.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2020-06-24 09:59:27 -07:00
Martí Bolívar 5abde2024f manifest: fix _ManifestImportDepth() call
We should be passing a str filename here. Caught by mypy.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2020-06-24 09:59:27 -07:00
Martí Bolívar b613c5d5ea manifest: clean up Project.posixpath logic
The posixpath() property if checks are copy/pasted from abspath, but
they aren't quite right. The _posixpath attribute really cares about
whether self.abspath is None, not self.topdir. Fix it.

This shouldn't affect correctness because self.abspath is non-None
exactly when self.topdir is, but is cleaner readability wise and will
make it easier to type annotate this property.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2020-06-24 09:59:27 -07:00
Martí Bolívar 743e3efe13 manifest: remove some ManifestProject properties
These are just duplicating functionality already present in the parent
class. The abspath property has to stay because, unlike Project,
ManifestProject might have a self._path set to None (... which,
looking at it again, is yet another reason it's not sensible to treat
it like a Project).

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2020-06-24 09:59:27 -07:00
Martí Bolívar d0dd0584ed manifest: fix Project.is_cloned() if abspath is None
The os.path.isdir() function errors out if you pass it None.
Caught by mypy.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2020-06-24 09:59:27 -07:00
Martí Bolívar 1251612484 manifest: re-work ManifestProject's fake attributes
Change the url, revision, and clone_depth properties to plain old
attributes, changing url from None to the empty string. This will be
used to make it possible to type annotate Project in a subsequent
commit.

This seems like even more evidence that ManifestProject is not a
Project in a subtyping sense, should not have been treated like one,
and should be removed (#327).

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2020-06-24 09:59:27 -07:00
Martí Bolívar 1edc4511d2 manifest: validate: fix error handling
We should be passing the actual version string to
ManifestVersionError, not the parsed version object.
Do the same in the f-string just for consistency.

Caught by mypy.

Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
2020-06-24 09:59:27 -07:00