Adjust the default verbosity by the amount implied by the number of
--verbose arguments given to west, for built-in commands.
This intermediate commit will be reworked later when main.py removes
its west.log calls. It's here for bisectability while we're reworking
other modules to use WestCommand I/O methods instead of west.log.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This is basically a duplicate of the west.log interface, except with:
- better names
- no use of global state
- support for a 'quiet' mode
The goal is to move over built-in commands to using this interface in
order to eliminate global state. That in turn may help us with our
goals of running more west tests in parallel and is just better design
anyway since the west APIs should be clean when used as a library
outside of any command.
We can move over extension functions in zephyr etc. over time as well,
probably also by using a helper that can detect older versions of west
and work anyway.
That will then allow us to deprecate west.log, removing it in the long
term, and adding support for a global "west --quiet <command> args"
style of invocation for the folks who just hate terminal output. I
want to get this conversion all done by the time Zephyr LTS3 is done,
so it's time to get the ball rolling now. I missed the boat on LTS2
and that makes me sad.
Part of: #149
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
The dataclass feature was added in Python v3.7, so we couldn't use it
while west still supported v3.6. West now requires v3.8 or later, so
we can use it.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Upstream Zephyr has moved to python v3.8 as a minimum version, so it's
OK for west to move too. Make that happen.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Extension commands can fail to load early in the process, when we load
the extension command information from the file system. We are not
currently handling this in a nice way, resulting in tracebacks instead
of sensible error messages.
Fix this by catching ExtensionCommandError and using similar error
handling as we are already doing from within run_command().
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Be clearer about what happened when we lack permissions to load the
manifest, but we need to do so in order to proceed. This can happen,
for instance, if it is owned by another user and read access is
denied.
Fixes: #579
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Move a few things around to group the manifest-related exception
handling together a bit better. This will make a future patch cleaner.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Commit 2327610945
("manifest: ManifestImportFailed: relax argument specification")
changed the attributes in ManifestImportFailed instances, but the
corresponding error handling code in main.py was not updated.
This is incorrect and results in an ugly traceback instead of a
properly formatted error message.
Restore the preferred behavior by using the new attribute 'imp'.
Fixes: #588
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Commit e8080e9c3f
("manifest: simplify and fix get_projects()") caused a regression in
the west command line interface, which turns out to be due to a bug in
main.py. The fix itself seems fine.
The bug is in main.py's set_zephyr_base() function, where we search
for any project named or with path 'zephyr' like this:
projects = manifest.get_projects(['zephyr'])
This was always incorrect: when searching for a project by path,
we cannot make any assumptions about the current working directory.
This line of code happens to work if you are running west from the
workspace topdir, which is what we do in testing, but it fails if you
are somewhere else in the file system, e.g. in WEST_TOPDIR/zephyr.
We happened to get bizarrely lucky before this fix to get_projects(),
because prior to that fix, we were comparing
Path(THE_MANIFEST_REPOSITORY_PATH).resolve()
with
Path('zephyr').resolve()
in the get_projects() call from set_zephyr_base().
As long as THE_MANIFEST_REPOSITORY_PATH is 'zephyr' (which it is by
default when using the upstream zephyr repository as the manifest
repository), this comparison will always result in get_projects()
returning the ManifestProject no matter where you are on the file
system, since we're comparing the same things to each other.
But *with* that fix in manifest.py, we need to also fix main.py to
pass a fully resolved WEST_TOPDIR/zephyr path to get_projects() when
searching by path, since what we are trying to do is find a project
with either name 'zephyr' or path 'zephyr' inside the current
workspace.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Commit c757d6650f
("configuration: add Configuration class") was part of a general
re-work of how configuration file handling went.
As part of this change, an internal _whence() method was added, which
returns a list of configuration file paths on the file system for a
given ConfigFile enumerator. _whence() is currently erroring out with
a RuntimeError() when a local configuration file is requested, but not
found.
This breaks error handling in some cases.
For example, consider a flow like this:
# This command fails for some reason, leaving a .west
# directory but no .west/config
$ west init
# This command bombs out with RuntimeError: a topdir is
# found, so main.py tries to create a Manifest, which asks
# for the 'manifest.path' configuration option from the
# local file, which ... boom
$ west list
It would be better to raise MalformedConfig from here instead. This
lets higher layers handle this error better. In the above case,
we now get this instead:
$ west list
FATAL ERROR: can't load west manifest
local configuration file not found
This is clearly better. There are probably still some other error
handling edge cases that aren't being handled properly as a result of
this change, but I'd be curious to know how many of them this fixes.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
It makes more sense to define the MalformedConfig exception from the
west.configuration module. Move it there. We'll use it in a subsequent
patch from the same west.configuration module. (This is currently not
possible without a circular dependency because west.manifest imports
west.configuration.)
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Without a file argument, we should still use fall_back=True to search
for the workspace in the environment. This behavior was removed in the
recent rework of path handling, introducing a regression where the
manifest repository cannot be found when outside the workspace. Fix
it.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
(cherry picked from commit d275142374)
This is an API break. We are still allowed to do that until west 1.0.
The west.manifest.Manifest class's path handling has become
unmaintainable and we need to overhaul it. The from_data() and
from_file() constructors are too clever and try to allow too many use
cases, which results in the guts of the manifest construction code
often not being able to tell whether it's in a real workspace or not.
This makes it difficult to know when we can safely read configuration
files or not. That in turn makes it difficult to accurately
distinguish between the 'manifest.path' value in the local configuration
file and the 'self: path:' value in the manifest data, especially when
we can take keyword arguments that say "hey, pretend this is your
path".
Additionally, there are a few assumptions in the code that pretend
that the manifest file itself always lives in the top level directory
of the manifest repository. That was a valid assumption up to the
point the 'manifest.file' configuration variable was introduced, but
it's no longer valid anymore, since 'manifest.file' can point to a
subdirectory of the manifest repository. This leaves us with some
hacks to find the git repository we live in that shouldn't be
necessary and which we can now remove with this overhaul.
Rework the whole thing so that we can correctly handle the following
situations:
- when running in a workspace, you should now use from_file() or
a newly introduced from_topdir() to make your Manifest
- when running outside of any workspace, use from_data()
From now on, we forbid creating a manifest from data "as if" it were
in a real workspace. If all you have is data, the abspath attributes
will all be None, reflecting reality.
Accordingly, rework the Manifest.__init__() method to either take a
topdir or a bit of data, but not both. Additionally, now that we have
both manifest.path and manifest.file configuration options, it's not
usually the right thing to ask for a Manifest from a *file*: what you
really usually want is a Manifest from a *workspace*, and for that you
just need a topdir. From the topdir, you can create a
west.configuration.Configuration, and from there you can read
manifest.path and manifest.file to get the complete path to the top
level manifest within its repository. For that reason, we remove the
source_file keyword argument from __init__() in favor of topdir and a
new 'config' argument.
For backwards compatibility, it's still allowed to use from_file() to
load another manifest file than the one pointed to by
topdir/manifest.path/manifest.file. However, this handling code is now
just a bit of special case trickery in from_file(), and it also
introduces a restriction that the other file is still in a git
repository in the workspace. This moves potential sources for error or
confusion out of Manifest.__init__() and the functions it calls. (This
has proven to be a useful feature in practice; I am aware of west
extensions that use it to compare the parsed contents of two different
manifest files within the same workspace.)
To migrate away from the now-outdated from_file(), introduce a new
from_topdir() factory method and use it throughout the tree. This is
clearer and more accurate code, which always avoids the hacks left
behind for compatibility in from_file().
Finally, add various new instance attributes to the Manifest class
that will let us tell the YAML path from the actual path, and the path
to the manifest file from the path to the manifest repository in the
workspace. These in turn let us remove some hacky code from the 'west
list' implementation, and allow us to rework our internals so that
ManifestProject is just a legacy remnant at this point, further
helping along with #327.
Keep tests up to date, removing obsolete code as needed and updating
cases to reflect API breakage.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Commit 3f81eb4195
("manifest: project names must be unique") fixed an issue where
projects with duplicate names were previously permitted, but the test
case file that was meant to catch regressions was incorrectly left
empty. Fix it.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Adjust some comments, docstrings, and logging calls, and do some
renaming. No functional changes expected; this will cut down on
clutter in the diffs in later patches.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Continue to accept all arguments previously accepted, but widen the
allowed argument types as follows:
- Allow passing a None argument as the project to signal the manifest
repository is where the import is happening. This will be useful for
allowing us to continue using this exception type without mandating
use of the ManifestProject class, which we'd like to get rid of
eventually.
- Generalize the 'filename' argument to allow specifying arbitrary
import data which could not be imported. This will be useful for
specifying failures that happen when importing a map by giving the
entire map that failed to import, instead of just a file name.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
It will be convenient in a future patch for west.manifest.validate()
to always return the validated dict which is loaded from YAML if all
we have is a str, so extend the behavior of the function by returning
the input dictionary it receives, or returning the dictionary it
parses.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
There's no reason to print the name; it's always "manifest".
And even if we were going to print the name, it should have been
quoted, so anyway this behavior is wrong.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
All built-in west commands are using the new WestCommand.config state
to access the configuration values. Extension commands which are
reading configuration values should migrate to using this interface as
well, as should any other out of tree users of the west.configuration
API.
Deprecation, rather than removal, is necessary since 'west build' and
other important Zephyr extensions rely on the existing API.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Add a 'config' property to WestCommand. This is a
west.configuration.Configuration accessible for the duration of
do_run(). For the same reason as the 'WestCommand.manifest' property
exists the way it does, we cannot add a new constructor argument to
capture the configuration. Handle this as a kwarg to run().
Take a Configuration in extension_commands().
Handle the necessary changes in main.py. These changes should be
invisible to all the extension commands I am aware of in the wild.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
The requires_installation constructor kwarg and property have been
deprecated for some time now, and they aren't in use in either Zephyr
v1.14 LTS or v2.7 LTS. It should be fine to remove them at this point.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Switch configuration file parsing to use a Configuration object.
This prserves existing functionality, but allows us to start more
easily passing around the configuration to other places that might
want it pre-loaded.
Retain existing behavior for backwards compatibility, but move the
initialization of the global west.configuration.config object over to
using the new API.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This is a high-level class that lets you load and interact with the
configuration files in an object-oriented way. It's similar to how
west.manifest.Manifest lets you interact with the manifest, but for
configuration files.
The new API uses 'config.get("some.option")' style methods instead of
the configparser style 'config.get("some", "option")' with a separated
section and key. This makes the code match the options as they are
documented, making it easier to read and grep for.
Having an object around will allow us to deprecate the current
implementation, which relies on global state. It will also make it
easier to override default configuration behavior in certain
circumstances that will be useful in later patches.
Part of the road towards resolving #149 and at least one other issue.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Add a kwarg that makes _location() skip the workspace search if the
local configuration file path is requested.
No behavioral changes expected. This is prep work for reusing this
helper in a new configuration API.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Tweak some incidental details and add a couple of extra asserts.
Prep work for adding another test later on.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
The argparse module changed some implementation details in in a way
that's causing a false failure in testing. Handle it.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
If a project has created a tag or branch with the name HEAD (which
they really shouldn't do) it will cause the command:
'git rev-parse --abbrev-ref HEAD' to fail to provide a ref.
Check if 'git rev-parse --abbrev-ref HEAD' fails to return a value and
if so do a `log.die()` rather than failing at the 'git merge-base'
stage with a less useful error.
Closes#561
The addition of project userdata was a breaking change for older
versions of west. This means we should have bumped the schema version
when it was added, but this was missed. Do it now.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This isn't working properly because the ManifestProject's revision is
"HEAD", so it looks like an ordinary project and we try to get
manifest-rev from it. That is obviously incorrect. Fix it by detecting
the ManifestProject explicitly and doing the right thing there.
Fixes: #550
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
We no longer have access to 3.6 in the system that provides our python
interpreter, so we'll have to do without it in CI.
I think in practice this is going to mean that 3.7 is the lowest
version of Python that west supports, but that's really a Zephyr
decision and I don't feel comfortable make it unilaterally here. So
for now we'll continue to treat failures on 3.6 as bugs, but we won't
test on that platform in CI. That's not ideal and I'll try to do
release testing on 3.6 for as long as I can, as it's the best we've
got.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This reverts commit 56ddb937a0.
Parallel updates are a great feature, but they break color.ui and
attempts to fix that have caused unexpected exceptions to be thrown on
Windows, so we need to back the feature out for now until we can get
it done in a way that works with both color and Windows.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This reverts commit 14b27b0940.
Parallel updates are a great feature, but they break color.ui and
attempts to fix that have caused unexpected exceptions to be thrown on
Windows, so we need to back the feature out for now until we can get
it done in a way that works with both color and Windows.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This reverts commit b1350a668c.
Parallel updates are a great feature, but they break color.ui and
attempts to fix that have caused unexpected exceptions to be thrown on
Windows, so we need to back the feature out for now until we can get
it done in a way that works with both color and Windows.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This reverts commit 040dd57ad0.
Parallel updates are a great feature, but they break color.ui and
attempts to fix that have caused unexpected exceptions to be thrown on
Windows, so we need to back the feature out for now until we can get
it done in a way that works with both color and Windows.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This reverts commit c851664e04.
Parallel updates are a great feature, but they break color.ui and
attempts to fix that have caused unexpected exceptions to be thrown on
Windows, so we need to back the feature out for now until we can get
it done in a way that works with both color and Windows.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Martí Bolívar