This prints performance statistics. Zephyr wants it for figuring out
slowdowns observed in CI.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
At this point, the manifest schema is frozen, so we can also update
west.manifest.SCHEMA_VERSION to 0.7, as this version of west parses
anything that 0.7 will be able to parse.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
"Close the parentheses" when done resolving a project import, and add
project revision information when importing from a project.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This kind of verbose detail about the parsed manifest is not always
useful to others.
E.g. downstream in the nRF Connect SDK, this is cluttering up the
output of a west extension that compares mainline zephyr's manifest
repository revisions with NCS's. This seems like an annoyance to me,
so this logging shouldn't be global.
We do, however, need it for debugging when resolving the manifest
and running 'west -vv update'.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
I keep running into edge cases trying to get proper import loop
detection going, so just catch the RecursionError it causes and
convert it to ManifestImportFailed.
Testing this takes a while, so we only will do that from CI for now.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
I want to add more values to the private context we pass around as we
resolve the manifest, but it's annoying to do so one parameter at a
time, since it requires updating a lot of functions.
Let's make a container object for this state and pass it around instead.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Track which projects we've already updated via update_importer() and
don't do it twice if it's already done.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Relax the restriction that west update cannot update individual
projects when manifest imports are in use, as long as the projects on
the command line are all defined in the manifest repository.
That's a well-defined enough operation to support, since this type of
project is basically equivalent to being defined in the top level
west.yml.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
From a CLI user perspective this means the 'west update' behavior is
unchanged with respect to project remote names from the west 0.6.x
series.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Do not restore a full-fledged Remote class, but allow projects to
specify the name of the remote that should be used when cloning them,
and respect it in project.py.
From an API perspective this preserves the breaking change in 630f069,
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Add tests for west init and west update for:
- simple zephyr downstream with revision fixed to a tag
- zephyr downstream with tag revision and project fork
- downstream of a manifest repository which has a directory
of submanifest files
- downstream of a zephyr rolling release
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Remove boilerplate by adding a manifest_repo fixture and cleaning up
some unnecessary string creations. Rename the fs_topdir fixture to
tmp_workspace to more clearly describe what it is.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
These verify the general behavior of blacklist and whitelist map keys,
and make sure they propagate through nested (self) imports.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Introduce a github action that will run tox on a number of python
versions and host OSes for testing.
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
As is already done for projects, pass the filter function implied by
an import's whitelists and blacklists down the call stack. When we
encounter a new filter due to a map import, compose the existing
filter with the new one. We use None as a signal that there is no
filter in place to avoid calling unnecessary functions.
The motivation behind this patch was to correctly log project addition
exactly once, when they are first discovered in the recursion tree.
However, it's actually necessary for correctness, and fixes a bug that
can be shown by example:
west.yml:
manifest:
...
self:
import:
file: foo.yml
name-whitelist: just-this-project
foo.yml:
manifest:
...
self:
import: bar.yml
bar.yml:
manifest:
...
projects:
- name: just-this-project
- name: but-not-this-one
Above, we should ultimately import just-this-project from bar.yml, and
not but-not-this-one. This implies propagating and ANDing together
filters implied by whitelists and blacklists from higher levels in the
tree.
This patch also includes a test case for this type of situation.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Pass the "known projects" map to the Manifest constructor via a kwarg.
This makes it possible to track most of the known projects when
resolving imports. That in turn partially resolves#360, except in the
case of import maps, where we are still creating a whole new manifest
and then filtering it out. We can fix that next.
This implies a hacky change to _check_paths_are_unique(): we need to
ignore checks involving the manifest repository if we're not top
level, as they may lead to false positives. In my opinion, this is yet
another good reason to avoid mixing the manifest repository itself
into the projects list.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
And add a shim in main.py that enables verbose debugging output if -v
is given and the manifest is re-parsed after the initial attempt is
made. (This makes commands like 'west update' and 'west manifest' emit
output when necessary.)
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This feature is an artifact of an abandoned design for manifest
imports which now has unclear utility. Rip it out for now until we are
sure we need it and have thought it through.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Convert TypeError to MalformedManifest to avoid a stack trace, and
improve the error message.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
It's nicer for the user of the manifest library (e.g. 'west diff') to
decide what should happen if git is not available, so move some
warning calls about missing git into project.py, and add a new helper
for dealing with dying when git is not found.
This lets us keep using log.wrn() for this detection, which we are
about to stop doing from manifest.py.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
The manpage for git merge-base --is-ancestor says the return value
should be 0 or 1. Anything else is probably an actual error, which we
should propagate up.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
As a WestCommand, this context knows its topdir. Passing this to
Manifest.from_file() is enough to avoid shelling out to git when
loading the manifest again, which we should do.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
The linter finishes in a few seconds and its errors can be addressed
quickly, whereas the test suite takes a bit longer to run. Let's do
the fast thing first.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This relates to issue #38, the TL;DR of which is that we've never
gotten around to properly separating west's application internals from
its API in source code, and instead relied on documentation to spell
out exactly what the API was that users could rely on.
Let's start fixing that by moving everything users can't rely on into
a new west.app. This includes everything in west.commands except the
__init__ module, so we can just make that a new src/west/commands.py
file and have it be a module instead of a package. This lets its
pykwalify schema file, west-commands-schema.yml, sit next to it in
src/west, which is flatter than before.
The code changes in this commit (source lines changed, rather than
files just getting moved around) are:
- change the entry point in setup.py to west.app.main:main
- change some imports in src/west/app/main.py to import from
west.app instead of west.commands
- add a new src/west/app/__init__.py, since we're not using
namespace packages
- changes in MANIFEST.in and test_help.py to reflect new paths
- adjust some comments and docstrings
This change makes the API divide clearer. This in turn exposes some
problems with our use of west.log from west.manifest:
1. logging to stdout is a bad practice from a library
2. west.log also uses global state (which relates to #149 also)
which non-command-line users won't have set up properly
(this is an example of why #1 is true)
Subsequent commits will move to only using west.log from within
west.app.* and the existing deprecated west.build and west.cmake APIs,
which users should be migrating away from anyway.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This is a SQLite database of coverage data now generated by default
during testing by pycov.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This completes the feature set for west manifest imports. We include a
test of name whitelists from the documentation as a basic sanity
check. Further testing is left to future work.
To close out this feature after this, we should only need more error
handling and testing. In particular, beyond general coverage issues,
we need to try to reject any obvious import cycles, and add tests for
the same.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Add additional line coverage for manifest.py using output from pycov.
I'm pretty happy with the resulting level of statement coverage (94%)
and the number of bugs that got shaken out by writing these (5 real
ones along with a couple of other improvements).
The remaining 6% is mostly trivial or part of the unimplemented
functions needed to import maps. The trivial stuff can get addressed
some other day if we're interested in getting to 100%, but I think
it's time to move on to implementing map imports after this, now that
we've got better test coverage for the manifest API in general and its
basic import feature in particular.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
We need to validate the imported data before assuming it's a
dictionary we can start doing things with. Fix that by calling out to
validate().
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
We need to be a bit more careful with our types.
For example, the string "not-a-manifest" can be loaded as YAML, and
evaluates to... itself. And the string 'manifest' sure is in the
string 'not-a-manifest', so the "'manifest' not in data" check in
validate() doesn't catch the invalidity.
Then we try to index ['manifest'] in a string, which doesn't work,
because string indexes must be integers. Boom.
Fix it with extra type checking.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Remove some dead code discovered with coverage testing:
- _import_from_project() is never called with a falsy import value.
Add an assert to preserve the invariant.
- _quote_maybe() and _west_yml() are leftovers that should have been
rebased out
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
The expected behavior for west commands imported from self is that
they will be added to the top level manifest, but this isn't working
due to some incorrect string/list conversions. Fix that up.
The same behavior should be extended to west command inheritance from
projects. We do allow some of that, but it's currently only bringing
in the first west-commands found in a submanifest if the project
doesn't specify commands of its own. Make the behavior more closely
match that of self-imports, except that in the case of projects, the
import order is 'reversed': parent project west-commands come before
inherited ones in order.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Improve test coverage for when west init ought to die with an error
because it's being asked to re-bootstrap something that's already
initialized.
- bootstrap init must not initialize a subdirectory that's already
- subprocess failure is not enough: we need to see 'already
initialized' in the error output, or it could be some other failure
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
West init's bootstrap() isn't smart enough about checking whether the
target directory has already been initialized. Fix that.
Also catch a shutil.Error and print a sensible message instead of
dumping stack.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This new pool uses large nodes and suitable for anything that does not
run sanitycheck. The queue for this pool is not as a long as with the
2XL pool which will give west instant results instead of waiting for 2XL
node to finish sanitycheck.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
We used to require the use of namespace packages, because west was
split into a bootstrapper and per-installation clone. That hasn't been
true since 0.6.0, so stop treating west like a namespace package.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Martí Bolívar