As a WestCommand, this context knows its topdir. Passing this to
Manifest.from_file() is enough to avoid shelling out to git when
loading the manifest again, which we should do.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
The linter finishes in a few seconds and its errors can be addressed
quickly, whereas the test suite takes a bit longer to run. Let's do
the fast thing first.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This relates to issue #38, the TL;DR of which is that we've never
gotten around to properly separating west's application internals from
its API in source code, and instead relied on documentation to spell
out exactly what the API was that users could rely on.
Let's start fixing that by moving everything users can't rely on into
a new west.app. This includes everything in west.commands except the
__init__ module, so we can just make that a new src/west/commands.py
file and have it be a module instead of a package. This lets its
pykwalify schema file, west-commands-schema.yml, sit next to it in
src/west, which is flatter than before.
The code changes in this commit (source lines changed, rather than
files just getting moved around) are:
- change the entry point in setup.py to west.app.main:main
- change some imports in src/west/app/main.py to import from
west.app instead of west.commands
- add a new src/west/app/__init__.py, since we're not using
namespace packages
- changes in MANIFEST.in and test_help.py to reflect new paths
- adjust some comments and docstrings
This change makes the API divide clearer. This in turn exposes some
problems with our use of west.log from west.manifest:
1. logging to stdout is a bad practice from a library
2. west.log also uses global state (which relates to #149 also)
which non-command-line users won't have set up properly
(this is an example of why #1 is true)
Subsequent commits will move to only using west.log from within
west.app.* and the existing deprecated west.build and west.cmake APIs,
which users should be migrating away from anyway.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This is a SQLite database of coverage data now generated by default
during testing by pycov.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This completes the feature set for west manifest imports. We include a
test of name whitelists from the documentation as a basic sanity
check. Further testing is left to future work.
To close out this feature after this, we should only need more error
handling and testing. In particular, beyond general coverage issues,
we need to try to reject any obvious import cycles, and add tests for
the same.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Add additional line coverage for manifest.py using output from pycov.
I'm pretty happy with the resulting level of statement coverage (94%)
and the number of bugs that got shaken out by writing these (5 real
ones along with a couple of other improvements).
The remaining 6% is mostly trivial or part of the unimplemented
functions needed to import maps. The trivial stuff can get addressed
some other day if we're interested in getting to 100%, but I think
it's time to move on to implementing map imports after this, now that
we've got better test coverage for the manifest API in general and its
basic import feature in particular.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
We need to validate the imported data before assuming it's a
dictionary we can start doing things with. Fix that by calling out to
validate().
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
We need to be a bit more careful with our types.
For example, the string "not-a-manifest" can be loaded as YAML, and
evaluates to... itself. And the string 'manifest' sure is in the
string 'not-a-manifest', so the "'manifest' not in data" check in
validate() doesn't catch the invalidity.
Then we try to index ['manifest'] in a string, which doesn't work,
because string indexes must be integers. Boom.
Fix it with extra type checking.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Remove some dead code discovered with coverage testing:
- _import_from_project() is never called with a falsy import value.
Add an assert to preserve the invariant.
- _quote_maybe() and _west_yml() are leftovers that should have been
rebased out
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
The expected behavior for west commands imported from self is that
they will be added to the top level manifest, but this isn't working
due to some incorrect string/list conversions. Fix that up.
The same behavior should be extended to west command inheritance from
projects. We do allow some of that, but it's currently only bringing
in the first west-commands found in a submanifest if the project
doesn't specify commands of its own. Make the behavior more closely
match that of self-imports, except that in the case of projects, the
import order is 'reversed': parent project west-commands come before
inherited ones in order.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Improve test coverage for when west init ought to die with an error
because it's being asked to re-bootstrap something that's already
initialized.
- bootstrap init must not initialize a subdirectory that's already
- subprocess failure is not enough: we need to see 'already
initialized' in the error output, or it could be some other failure
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
West init's bootstrap() isn't smart enough about checking whether the
target directory has already been initialized. Fix that.
Also catch a shutil.Error and print a sensible message instead of
dumping stack.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This new pool uses large nodes and suitable for anything that does not
run sanitycheck. The queue for this pool is not as a long as with the
2XL pool which will give west instant results instead of waiting for 2XL
node to finish sanitycheck.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
We used to require the use of namespace packages, because west was
split into a bootstrapper and per-installation clone. That hasn't been
true since 0.6.0, so stop treating west like a namespace package.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
The term "west installation" is inaccurate since 0.6, since we no longer
clone west into .west/west.
The term "workspace" is more familiar and seems to be more readily
understood, so switch to that instead. Keep backwards compatibility
with the requires_installation kwarg and attribute to WestCommand for
now.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Use f-strings instead of str.format() or string concatenation with +
whenever practical. Tweak a few log messages while we're here.
I may be getting a little f-string-happy, but I think having a single,
consistent style throughout the code base makes things easier to read,
and f-strings look like the natural default choice to me.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This also stops using it from Project.git(). We no longer have any
in-tree users, and neither Zephyr 2.1.x nor 1.14.x appear to have any
users either.
We can revert this commit later if it causes trouble.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Commit 639ce48 ("commands: project: consolidate error handling
behavior") did some refactoring and error handling but got one of the
error messages wrong. Fix it.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
The addition of f-strings to the Python language in 3.6 (which is now
our minimum version) makes using Project.format both harder to read
and slower to execute than necessary.
It's also just too magical when used implicitly from Project.git() or
one of its callers.
Replace with f-strings.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Add a remote repository to the setup we create in conftest.py whose
revision is a tag. Add testing for it in project.py, generalizing
update_helper() a bit while doing this to keep the code readable.
This adds coverage for basic west operations when manifest projects
have revisions which are tags.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Whoops. I should have thought of this when requesting this change.
When a revision is not a SHA, fetch it and update manifest-rev to it
in two steps, setting manifest-rev to FETCH_HEAD^{commit}.
This still only fetches the refs we need, but works for all types of
refs.
Fixes: #349
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Use the as_yaml() and as_frozen_yaml() convenience functions,
which is safe to do now that we don't get an OrderedDict out of
west.manifest.as_dict() and friend.
This requires PyYAML 5.1 or later.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This is a potentially breaking API change, as as_dict() and
as_frozen_dict() return plain dict instead of OrderedDict.
Now that west requires python 3.6+, we can rely on insertion ordered
dictionaries, at least on CPython (sorry, pypy).
This makes it possible to use yaml.safe_dump() directly on the dict
representation of a west.manifest.Manifest and get predictably ordered
results, without having to go through the extra step of using an
OrderedDict and thus having to add a representer for it in the yaml
library.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This is like as_frozen_dict(), except not frozen.
This will be used by west manifest --resolve.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Currently `west update` will fetch into refs/west ref space in order to
ensure that SHAs located on branches can be checked out.
This commit cleans the refs/west ref space after fetching to minimize
noise seen by users, as example when executing 'git log'.
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
Adjust west init to ignore imports. It doesn't need anything other
than what's available in the top level manifest.
Add an importer callback from west update, which can handle
resolution when project data is needed but not available. Move the
_update() private helper to an instance method. This helps us keep
more state inside of self, which is useful now that we want to invoke
this method from an instance-specific callback.
This imposes some limitations when west update is used with manifest
imports. In particular, it is not permitted to 'west update
SOME_PROJECT' if the manifest uses imports.
The reason why is that it would require the west.manifest API to track
extra information during manifest resolution that it's not capable of
doing, and which would take more work to do than we think is
worthwhile right now.
One pathological example follows.
my-manifest/west.yml:
manifest:
projects:
# 'my-upstream' is the upstream manifest that defines its
# own projects. I want all of them. I want to track master.
- name: my-upstream
url: https://github.com/my-vendor/upstream-manifest
import: true
Suppose my-upstream points at commit A right now, i.e. "git rev-parse
refs/heads/manifest-rev" prints "A", but the remote fetch URL
https://github.com/my-vendor/upstream-manifest has a master commit
pointing at B.
Further, suppose we have:
my-upstream/west.yml at commit A:
manifest:
# ...
projects:
- name: p1
my-upstream/west.yml at commit B:
manifest:
# ...
projects:
- name: p2
Think about what would happen if the user runs:
$ west update
In this case, my-upstream is updated, and therefore its manifest-rev
should now point at commit B. This introduces project p2, which is not
cloned on the local file system. So p2 should be fetched.
The p1 project is 'abandoned': it still exists on the local file
system, but west forgets about it entirely unless we do something like
set my-upstream's revision to point back at A.
OK, now what if we run:
$ west update p2
Given the above, this is a seemingly sensible request: if plain 'west
update' would clone and update p2, then 'west update p2' should also
work, no? Or should it?
If yes, that would mean west would need to speculatively update all
projects that have import statements, hunting for a p2 project. We
can't do that right now, and it's not clear how such a thing would be
defined.
Or what about this?
$ west update p1
If plain 'west update' means p1 is no longer a project, this command
should fail, no? Or should it succeed?
If it should fail, that would require west to fetch from
my-upstream (since that's the source of the manifest data that defined
p1 in the first place) and check if it's still there, and fail when
it's not.
If it should succeed, why would 'west update' with no arguments
produce entirely different results?
The presence of pathological examples like these is too hard to deal
with for an initial implementation, so we're going to intentionally
limit 'west update' to take no arguments when manifest imports are
used.
Let's see if people care, and if they do, how they would propose
resolving situations such as the one above.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Initial limitation is:
Suggested-by: Carles Cufí <carles.cufi@nordicsemi.no>
Add some logic to detect when we couldn't load the manifest because it
contains an import that we can't resolve. Select a subset of built-in
commands that should be runnable anyway, and let those proceed (west
update currently can't handle this, but we'll fix that next), but
otherwise fail.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Add the API-level features required to implement manifest imports of
simple files, directories, and sequences of these.
Defer implementation of maps as import keys and import loop detection
for now. There's already quite a bit to test here. We'll add the rest
in follow-up commits, once we've fleshed out the built-in commands.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Any project element or the self map can contain import: keys. Each
import: key specifies how west should locate additional manifest files
inside the project or manifest repository and import their contents
into the final manifest.
The import: key values may be any one of:
- a boolean
- a string which is a relative path to a manifest file or directory
containing zero or more manifest files with .yml extensions
- a map which specifies project data to import
- a sequence of any of the above
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Refactor the validation internals to add a separate helper, which can
be called to do syntactic validation without actually resolving the
manifest.
This is just a convenience for users in case they don't want to bother
resolving an entire manifest and just want to do a quick schema check.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
The fact that manifest.path is a thing that gets set at init time is
actually pretty important for people to understand. Don't make it a
debug item.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Try to avoid parsing the manifest from west init unless we really need
to, and move all the ZEPHYR_BASE logic into main.py.
We'll need to continue to try to parse the manifest when bootstrapping
in order to figure out manifest.path, but we won't be able to resolve
imports, so it's best to avoid manifest resolution unless the whole
thing really needed.
We have to move the ZEPHYR_BASE logic out for a similar reason: the
zephyr project may not appear anywhere in the top level manifest file,
which is all that's accessible to west init.
This changes the zephyr.base behavior slightly: west will attempt to
set the config option "forever", i.e. any time an extension is run and
it's not set, and not just during west init. That's OK though -- for
Zephyr installations, which is what count for now, that means it only
happens once.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Include the revision. This is useful enough when debugging to want to
have it. Update the quoting behavior so it's more directly eval()-able.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Only print tracebacks at -vvv verbosity, but make sure fatal errors
are logged as such.
The simple -v is often used in extension commands to produce more
output, and shouldn't be printing information only of use when
debugging west itself. A similar comment applies to -vv.
Consolidate the CommandContextError handling into CommandError
handling. I want to remove CommandContextError entirely but I can't
while Zephyr LTS still lives. This is a first step.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
West 0.7.0 will require Python 3.6, just like Zephyr 2.2 will.
Move to the zephyrproject-rtos/ci container which contains this
Python. This has the ancillary benefit of making this test environment
the same as Zephyr's.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
The handling code here is cutting out a lot of useful content that's
added by the exception handler in main.py, so there's no point in
having it.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Martí Bolívar