Fix copy/paste issue in which the 'runs-on' was always ubuntu-latest so
we didn't actually do anything with the matrix.os to run on mac or
windows.
Also cleanup some minor formatting and report the sys.platform so we can
verify in the future we are on the correct platform.
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
The test_import_project*() cases in test_project.py are failing on
windows for reasons that are irrelevant and ultimately down to path
comparisons not being canonicalized for that platform.
Rather than fix tests I don't really think are broken, check if the
expected URL is an actual directory and fall back on PurePath
comparisons instead.
Keep the behavior the same for macOS and Linux.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
If the manifest URL is a windows-style path,
e.g. C:\Users\marti\my-manifest, then the current code produces a
manifest.path configuration option set to '\Users\marti\my-manifest'
instead of the expected 'my-manifest'.
Fix it by using PurePath so we get PureWindowsPath on windows, instead
of using the low-level posixpath. Note that PurePath is PurePosixPath
on POSIX platforms, so the behavior is the same on those.
Fix the import order while we are here, moving stdlib imports to their
proper section.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Git is returning empty content for dir\file.yml on Windows.
Using PurePosixPath fixes it there, and can't possibly break anything
on Unix.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Hopefully the last one.
Note that this "fix" is purely cosmetic, for use when reading debugger
output.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
The LocalPath object is not iterable on at least some versions of
pytest, which means that some checks done in subprocess ("\t" in
some_instance) are throwing exceptions.
Sprinkle some str() calls around to make this go away.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
The shlex.split() call done in Project.git() when you pass a string
isn't doing the right thing for Windows-style abspaths. Let's use
lists instead.
This converts all the cases caught by the test suite. Hopefully there
aren't any more.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This prints performance statistics. Zephyr wants it for figuring out
slowdowns observed in CI.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
At this point, the manifest schema is frozen, so we can also update
west.manifest.SCHEMA_VERSION to 0.7, as this version of west parses
anything that 0.7 will be able to parse.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
"Close the parentheses" when done resolving a project import, and add
project revision information when importing from a project.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This kind of verbose detail about the parsed manifest is not always
useful to others.
E.g. downstream in the nRF Connect SDK, this is cluttering up the
output of a west extension that compares mainline zephyr's manifest
repository revisions with NCS's. This seems like an annoyance to me,
so this logging shouldn't be global.
We do, however, need it for debugging when resolving the manifest
and running 'west -vv update'.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
I keep running into edge cases trying to get proper import loop
detection going, so just catch the RecursionError it causes and
convert it to ManifestImportFailed.
Testing this takes a while, so we only will do that from CI for now.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
I want to add more values to the private context we pass around as we
resolve the manifest, but it's annoying to do so one parameter at a
time, since it requires updating a lot of functions.
Let's make a container object for this state and pass it around instead.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Track which projects we've already updated via update_importer() and
don't do it twice if it's already done.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Relax the restriction that west update cannot update individual
projects when manifest imports are in use, as long as the projects on
the command line are all defined in the manifest repository.
That's a well-defined enough operation to support, since this type of
project is basically equivalent to being defined in the top level
west.yml.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
From a CLI user perspective this means the 'west update' behavior is
unchanged with respect to project remote names from the west 0.6.x
series.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Do not restore a full-fledged Remote class, but allow projects to
specify the name of the remote that should be used when cloning them,
and respect it in project.py.
From an API perspective this preserves the breaking change in 630f069,
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Add tests for west init and west update for:
- simple zephyr downstream with revision fixed to a tag
- zephyr downstream with tag revision and project fork
- downstream of a manifest repository which has a directory
of submanifest files
- downstream of a zephyr rolling release
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Remove boilerplate by adding a manifest_repo fixture and cleaning up
some unnecessary string creations. Rename the fs_topdir fixture to
tmp_workspace to more clearly describe what it is.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
These verify the general behavior of blacklist and whitelist map keys,
and make sure they propagate through nested (self) imports.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Introduce a github action that will run tox on a number of python
versions and host OSes for testing.
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
As is already done for projects, pass the filter function implied by
an import's whitelists and blacklists down the call stack. When we
encounter a new filter due to a map import, compose the existing
filter with the new one. We use None as a signal that there is no
filter in place to avoid calling unnecessary functions.
The motivation behind this patch was to correctly log project addition
exactly once, when they are first discovered in the recursion tree.
However, it's actually necessary for correctness, and fixes a bug that
can be shown by example:
west.yml:
manifest:
...
self:
import:
file: foo.yml
name-whitelist: just-this-project
foo.yml:
manifest:
...
self:
import: bar.yml
bar.yml:
manifest:
...
projects:
- name: just-this-project
- name: but-not-this-one
Above, we should ultimately import just-this-project from bar.yml, and
not but-not-this-one. This implies propagating and ANDing together
filters implied by whitelists and blacklists from higher levels in the
tree.
This patch also includes a test case for this type of situation.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Pass the "known projects" map to the Manifest constructor via a kwarg.
This makes it possible to track most of the known projects when
resolving imports. That in turn partially resolves#360, except in the
case of import maps, where we are still creating a whole new manifest
and then filtering it out. We can fix that next.
This implies a hacky change to _check_paths_are_unique(): we need to
ignore checks involving the manifest repository if we're not top
level, as they may lead to false positives. In my opinion, this is yet
another good reason to avoid mixing the manifest repository itself
into the projects list.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
And add a shim in main.py that enables verbose debugging output if -v
is given and the manifest is re-parsed after the initial attempt is
made. (This makes commands like 'west update' and 'west manifest' emit
output when necessary.)
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This feature is an artifact of an abandoned design for manifest
imports which now has unclear utility. Rip it out for now until we are
sure we need it and have thought it through.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Convert TypeError to MalformedManifest to avoid a stack trace, and
improve the error message.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
It's nicer for the user of the manifest library (e.g. 'west diff') to
decide what should happen if git is not available, so move some
warning calls about missing git into project.py, and add a new helper
for dealing with dying when git is not found.
This lets us keep using log.wrn() for this detection, which we are
about to stop doing from manifest.py.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
The manpage for git merge-base --is-ancestor says the return value
should be 0 or 1. Anything else is probably an actual error, which we
should propagate up.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
As a WestCommand, this context knows its topdir. Passing this to
Manifest.from_file() is enough to avoid shelling out to git when
loading the manifest again, which we should do.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
The linter finishes in a few seconds and its errors can be addressed
quickly, whereas the test suite takes a bit longer to run. Let's do
the fast thing first.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This relates to issue #38, the TL;DR of which is that we've never
gotten around to properly separating west's application internals from
its API in source code, and instead relied on documentation to spell
out exactly what the API was that users could rely on.
Let's start fixing that by moving everything users can't rely on into
a new west.app. This includes everything in west.commands except the
__init__ module, so we can just make that a new src/west/commands.py
file and have it be a module instead of a package. This lets its
pykwalify schema file, west-commands-schema.yml, sit next to it in
src/west, which is flatter than before.
The code changes in this commit (source lines changed, rather than
files just getting moved around) are:
- change the entry point in setup.py to west.app.main:main
- change some imports in src/west/app/main.py to import from
west.app instead of west.commands
- add a new src/west/app/__init__.py, since we're not using
namespace packages
- changes in MANIFEST.in and test_help.py to reflect new paths
- adjust some comments and docstrings
This change makes the API divide clearer. This in turn exposes some
problems with our use of west.log from west.manifest:
1. logging to stdout is a bad practice from a library
2. west.log also uses global state (which relates to #149 also)
which non-command-line users won't have set up properly
(this is an example of why #1 is true)
Subsequent commits will move to only using west.log from within
west.app.* and the existing deprecated west.build and west.cmake APIs,
which users should be migrating away from anyway.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This is a SQLite database of coverage data now generated by default
during testing by pycov.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This completes the feature set for west manifest imports. We include a
test of name whitelists from the documentation as a basic sanity
check. Further testing is left to future work.
To close out this feature after this, we should only need more error
handling and testing. In particular, beyond general coverage issues,
we need to try to reject any obvious import cycles, and add tests for
the same.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Add additional line coverage for manifest.py using output from pycov.
I'm pretty happy with the resulting level of statement coverage (94%)
and the number of bugs that got shaken out by writing these (5 real
ones along with a couple of other improvements).
The remaining 6% is mostly trivial or part of the unimplemented
functions needed to import maps. The trivial stuff can get addressed
some other day if we're interested in getting to 100%, but I think
it's time to move on to implementing map imports after this, now that
we've got better test coverage for the manifest API in general and its
basic import feature in particular.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
We need to validate the imported data before assuming it's a
dictionary we can start doing things with. Fix that by calling out to
validate().
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
We need to be a bit more careful with our types.
For example, the string "not-a-manifest" can be loaded as YAML, and
evaluates to... itself. And the string 'manifest' sure is in the
string 'not-a-manifest', so the "'manifest' not in data" check in
validate() doesn't catch the invalidity.
Then we try to index ['manifest'] in a string, which doesn't work,
because string indexes must be integers. Boom.
Fix it with extra type checking.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Kumar Gala