Adjust west init to ignore imports. It doesn't need anything other
than what's available in the top level manifest.
Add an importer callback from west update, which can handle
resolution when project data is needed but not available. Move the
_update() private helper to an instance method. This helps us keep
more state inside of self, which is useful now that we want to invoke
this method from an instance-specific callback.
This imposes some limitations when west update is used with manifest
imports. In particular, it is not permitted to 'west update
SOME_PROJECT' if the manifest uses imports.
The reason why is that it would require the west.manifest API to track
extra information during manifest resolution that it's not capable of
doing, and which would take more work to do than we think is
worthwhile right now.
One pathological example follows.
my-manifest/west.yml:
manifest:
projects:
# 'my-upstream' is the upstream manifest that defines its
# own projects. I want all of them. I want to track master.
- name: my-upstream
url: https://github.com/my-vendor/upstream-manifest
import: true
Suppose my-upstream points at commit A right now, i.e. "git rev-parse
refs/heads/manifest-rev" prints "A", but the remote fetch URL
https://github.com/my-vendor/upstream-manifest has a master commit
pointing at B.
Further, suppose we have:
my-upstream/west.yml at commit A:
manifest:
# ...
projects:
- name: p1
my-upstream/west.yml at commit B:
manifest:
# ...
projects:
- name: p2
Think about what would happen if the user runs:
$ west update
In this case, my-upstream is updated, and therefore its manifest-rev
should now point at commit B. This introduces project p2, which is not
cloned on the local file system. So p2 should be fetched.
The p1 project is 'abandoned': it still exists on the local file
system, but west forgets about it entirely unless we do something like
set my-upstream's revision to point back at A.
OK, now what if we run:
$ west update p2
Given the above, this is a seemingly sensible request: if plain 'west
update' would clone and update p2, then 'west update p2' should also
work, no? Or should it?
If yes, that would mean west would need to speculatively update all
projects that have import statements, hunting for a p2 project. We
can't do that right now, and it's not clear how such a thing would be
defined.
Or what about this?
$ west update p1
If plain 'west update' means p1 is no longer a project, this command
should fail, no? Or should it succeed?
If it should fail, that would require west to fetch from
my-upstream (since that's the source of the manifest data that defined
p1 in the first place) and check if it's still there, and fail when
it's not.
If it should succeed, why would 'west update' with no arguments
produce entirely different results?
The presence of pathological examples like these is too hard to deal
with for an initial implementation, so we're going to intentionally
limit 'west update' to take no arguments when manifest imports are
used.
Let's see if people care, and if they do, how they would propose
resolving situations such as the one above.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Initial limitation is:
Suggested-by: Carles Cufí <carles.cufi@nordicsemi.no>
Add some logic to detect when we couldn't load the manifest because it
contains an import that we can't resolve. Select a subset of built-in
commands that should be runnable anyway, and let those proceed (west
update currently can't handle this, but we'll fix that next), but
otherwise fail.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Add the API-level features required to implement manifest imports of
simple files, directories, and sequences of these.
Defer implementation of maps as import keys and import loop detection
for now. There's already quite a bit to test here. We'll add the rest
in follow-up commits, once we've fleshed out the built-in commands.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Any project element or the self map can contain import: keys. Each
import: key specifies how west should locate additional manifest files
inside the project or manifest repository and import their contents
into the final manifest.
The import: key values may be any one of:
- a boolean
- a string which is a relative path to a manifest file or directory
containing zero or more manifest files with .yml extensions
- a map which specifies project data to import
- a sequence of any of the above
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Refactor the validation internals to add a separate helper, which can
be called to do syntactic validation without actually resolving the
manifest.
This is just a convenience for users in case they don't want to bother
resolving an entire manifest and just want to do a quick schema check.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
The fact that manifest.path is a thing that gets set at init time is
actually pretty important for people to understand. Don't make it a
debug item.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Try to avoid parsing the manifest from west init unless we really need
to, and move all the ZEPHYR_BASE logic into main.py.
We'll need to continue to try to parse the manifest when bootstrapping
in order to figure out manifest.path, but we won't be able to resolve
imports, so it's best to avoid manifest resolution unless the whole
thing really needed.
We have to move the ZEPHYR_BASE logic out for a similar reason: the
zephyr project may not appear anywhere in the top level manifest file,
which is all that's accessible to west init.
This changes the zephyr.base behavior slightly: west will attempt to
set the config option "forever", i.e. any time an extension is run and
it's not set, and not just during west init. That's OK though -- for
Zephyr installations, which is what count for now, that means it only
happens once.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Include the revision. This is useful enough when debugging to want to
have it. Update the quoting behavior so it's more directly eval()-able.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Only print tracebacks at -vvv verbosity, but make sure fatal errors
are logged as such.
The simple -v is often used in extension commands to produce more
output, and shouldn't be printing information only of use when
debugging west itself. A similar comment applies to -vv.
Consolidate the CommandContextError handling into CommandError
handling. I want to remove CommandContextError entirely but I can't
while Zephyr LTS still lives. This is a first step.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
West 0.7.0 will require Python 3.6, just like Zephyr 2.2 will.
Move to the zephyrproject-rtos/ci container which contains this
Python. This has the ancillary benefit of making this test environment
the same as Zephyr's.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
The handling code here is cutting out a lot of useful content that's
added by the exception handler in main.py, so there's no point in
having it.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
An unambiguous __repr__ is now possible since the removal of the
Defaults and Remote API. This useful e.g. when printing lists of
Projects in a debugger.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Sphinx in the Zephyr documentation is complaining about some of these
"any" references due to ambiguity with method names from other
packages.
Disambiguate them.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
The main() function has gotten really long, and it's getting hard to
read. I want to break it up, but there's a fair bit of interdependent
state to manage.
Rather than using global variables, create a WestApp instance to
contain them all and use it instead. This helps with readability
without making #149 harder to do.
We'll extend this later to handle manifest imports.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Let's just get rid of this entirely to make it super clear that those
sections only have a meaning within the parsing of a single manifest
file, and are not relevant to the abstract representation of a
manifest.
This lets us rewrite the Project() constructor so it matches the tuple
representation in the manifest imports documentation quite closely.
This should hopefully make the API easier to use and understand.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Without this patch, you can't use 'west forall' on the manifest
project by using its name (manifest).
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
The code being tested is fine; there are just issues in the test cases
which make them fail on Windows.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
We are getting an extra newline at the end of our help, which is
inconsistent with how argparse usually works and is causing tox issues
on Windows.
Avoid having an empty line at the end of help messages.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
I'm writing the literal string too often. It will make fixing #249 (be
more forgiving about manifest file name) harder. Add a symbol.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Functional differences:
- Project and ManifestProject don't use __slots__ anymore
- you can no longer set ManifestProject.revision
- the Manifest remotes, defaults, get_remote() attributes/method are gone
- the Project remote attribute is gone
- frozen dictionaries have inlined URLs now, not remotes/defaults
These breaking changes are going to be needed for manifest importing.
We can't keep exposing defaults and remotes objects anymore, because
the semantics for project overrides mean that they aren't sensible
concepts at the API level.
Also save enough state in Project and ManifestProject to find out when
an attribute's data was explicitly part of the manifest file, or came
from a default. This will be useful when applying project overrides.
Update test_project.py and test_manifest.py to keep up with these
changes.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
The {remote_name} format string attribute is going away, so there's
not much better we can do. We don't use this, anyway.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This prevents us from dumping stack on a broken pipe.
Depending on when the exception occurs, we may still see something
like this:
Exception ignored in: <colorama.ansitowin32.StreamWrapper object at 0x7fc755d5c990>
BrokenPipeError: [Errno 32] Broken pipe
We can try to avoid even this in Python 3.8, which added a
sys.unraisablehook() that can be called instead. Let's defer that for
now, though; 3.8 hasn't hit my Arch installation yet and I want to
wait for that to start playing with it.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Upon review, we've decided to version the manifest files using a
separate-but-related namespace to west version numbers.
Summary of changes:
- If the manifest schema changes in west X.Y, that new manifest schema
version is also X.Y. So if west 0.8 adds new manifest features not
available in 0.7, west 0.8 supports manifest schema 0.8.
- If the manifest schema does *not* change in west X.(Y+1), the
manifest schema is the same as it was in X.Y. So if west 0.9 has the
same manifest schema as west 0.8, then west 0.9 still has manifest
schema 0.8.
This change lets us pin a single manifest schema version to each set
of changes, without sacrificing our ability to say "you need at least
west X.Y to support this manifest".
Suggested-by: Carles Cufí <carles.cufi@nordicsemi.no>
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Reformat these according to the style Sphinx seems to really want.
(I used an emacs package I just found for this.)
Hide some internal details in the west.commands package.
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
We used to use two newlines between definitions but then gradually
stopped doing that. Formalize that.
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
These were left behind in west after extensions were implemented and
'west build' and friends were made into extensions.
They're Zephyr-specific and have no business here, so mark them
deprecated.
We can't remove them though, because Zephyr v1.14 uses them and is
still supported. However, let those who enable python warnings know
they're on their way out.
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
Break Manifest._load() up into subroutines.
Adding support for manifest imports would make it too long to keep a
single function like this.
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
Make the project-specific errors easier to understand, and improve the
presentation of the results.
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
This requires a bit of special-casing depending on the exact command
to have the results at the CLI make sense on error.
Remove an unnecessary setter while we are here.
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
This refactoring will make it easier to handle ManifestVersionError.
Make 'west help' invoke a WestCommand too. API improvements made over
time make this easy to do.
It also has the pleasant side effect of making it really easy for
"west --help init" to be equivalent to "west help init" instead of
"west --help".
It works slightly different in case of error, though
("west --help unknown" produces different output than "west help
unknown"). I can't fix that without breaking the argparse abstraction
barrier, though, so it's OK.
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
Let's start using two spaces for our indentation.
This was chosen semi-arbitrarily to be consistent with how argparse
indents arguments and options.
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
As we evolve the manifest data, we want to be able to enforce a
minimum version of west required to parse it. Add an optional
'version' key to the data to make this possible.
We have to validate this before checking the manifest against the
schema, as later versions of west will likely extend the schema in
ways that would raise errors if we used our schema to check the data.
We allow the version to be parsed as a YAML float (like 1.0) and
convert it to a string as a convenience for the user.
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
It's convenient when using this API to be able to pass a string
containing YAML data instead of requiring the caller to load the YAML
data every time.
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
These should be separated into paragraphs for standard library,
third-party dependency, and internal imports respectively, but they're
not. Fix that.
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
GitHub user E3V3A reports that 'ProgramData' in the cygwin environment
corresponds to PROGRAMDATA in the parent environment, so we ought to
be able to use that to locate the system configuration file in a way
that is consistent with the outer native Windows shell expectations.
Fixes: #300
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
In certain situations, such as if the project remote is an empty
repository or otherwise prevents 'west update' from setting up the
manifest-rev branch, we can't freeze the manifest even though a
project has been cloned.
Handle that by raising RuntimeError, so that higher layers know what
went wrong and can print a sensible error message instead of dumping
stack.
Fixes: #314
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
We're getting increasing interest in west being its own tool, to be
used standalone from Zephyr. Unfortunately we are still too tightly
coupled for that to be possible, but the current behavior of erroring
out is not correct. Let's downgrade that to a warning and have the
warning print the current workaround.
Relates to: #246
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
Add a mandatory space between shell arguments. Fix the test cases so
they actually test this as advertised.
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
Every time ZEPHYR_BASE and west config clashed with each other it's been
because I moved across _several_ west installations and forgot to
adjust ZEPHYR_BASE. Fix the warning message so:
- it doesn't refer to the "west config" like there's only one!
- it shows which exact west topdir it got a west config from;
- it shows the actual value in west config and not a construction.
Before:
WARNING: ZEPHYR_BASE=/home/john/west1/zephyr in the calling
environment will be used, but was set to
/home/john/west2/zephyr in west config.
After:
WARNING: ZEPHYR_BASE=/home/john/west1/zephyr in the calling
environment will be used, but the zephyr.base config option in
/home/john/west2 is "zephyr" which implies
ZEPHYR_BASE=/home/john/west2/zephyr
When something goes wrong, the user unfortunately needs the "raw" /
low-level data; synthetized information only obscures what's going on
and may even be incorrect in case of a corner case / bug.
PS: the previous message was especially confusing when the current
directory includes some symbolic link(s) hence doesn't match anything in
the warning. In that (admittedly unsupported) case, pointing straight at
the specific west config helps even more.
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
Use the phrase "extension commands" and print the project path along
with its name.
We don't use {name_and_path} here because users probably aren't used
to that format yet.
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
This just prints the top level directory of the west installation.
To fit it into the 'west help' output, tweak the section headings.
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
We still need to update the manifest-rev ref even if we don't
fetch. D'oh. Hotfix the issue; a follow up cleanup should separate the
update to manifest-rev from the _fetch() command and move it to
_update() only.
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
Martí Bolívar