Adjust some comments and remove parentheses (which are unnecessary in
assert statements in Python).
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
The west.manifest module is not in great shape:
- Though it's possible to create a manifest from data, this still
looks for a topdir and a manifest.path configuration option on
disk. This is wrong and contrary to the goal that we can parse
manifest data independently of the file system.
You can see that this doesn't really make sense in the hacks that
the west init implementation needs to make things work, and the fact
that all the tests which operate on data only have to patch
west_topdir(), etc.
- The Project definition repeatedly finds the west topdir. Even
when this is necessary, it should only be done once, and the value
cached and passed around.
- The ManifestProject abstraction makes several bizarre semantic
decisions, some of which never made sense, some of which are
just stale leftovers from when it was SpecialProject.
- The manifest_path() helper mutates the global configuration object
Add some new kwargs to the Manifest factories and constructor, which
allow the user to avoid using west_topdir() or west.configuration when
they do not want to, and propagate the new information down into
Project and ManifestProject.
This allows users to parse manifest files and data without forcing
them to be inside an actual installation (and/or to parse other
manifest files than the one currently pointed to by manifest.path), at
the cost of not having abspath or posixpath attributes in some cases.
Fix up Project by taking a new topdir kwarg and saving it in a slot.
Fix up ManifestProject so that it generally makes more sense. This now
enforces an existing requirement in the documentation that no project
can be named "manifest", as that's reserved for the manifest
repository. Fix some other thinkos as well by deprecating the revision
and url kwargs, which are SpecialProject leftovers, and taking the
path as an argument as-is, even though that means it can be None now.
I've tried to preserve backwards compatibility whenever possible,
especially in situations where the user just wants to parse the
manifest in the current west installation based on file system
searches and configuration options.
The additional kwargs to Project and ManifestProject are breaking
changes for any code which was using this module for something else,
but the lack of sense that those made before probably meant such
code didn't do anything useful, so that's okay.
This also breaks compatibility with "west list zephyr" when run
outside of the top level directory -- you need to say "west list
manifest" or give the complete relative path to zephyr now.
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
Add a topdir argument to each of these functions:
- read_config()
- update_config()
- delete_config()
If set, then topdir/.west/config will be used as the local
configuration file if necessary. This allows users of this module to
decouple themselves from west.util.west_dir() without having to resort
to temporarily setting environment variables.
Keep the tests up to date, adding a couple of test cases to make sure
the new kwarg works as expected.
While we're here, re-work the fixture setup so we don't ever have to
monkey-patch west.configuration._location() by always pointing the
WEST_CONFIG_xxx environment variables inside the tmpdir we're running
each test case from. This lets us avoid some extra work in tox.ini and
makes the resulting test code easier to read, without sacrificing any
safety in terms of touching the user's actual files.
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
It's not necessary. It makes perfect sense to use west config --global
outside of any installation, for example.
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
The user-visible features in this point release are:
- an optimized west update via its new --fetch command line flag and
update.fetch config option. The default value, "smart", skips fetching
SHAs and tags which are available locally.
- better error-handling in the manifest, diff, status, forall, and
update commands.
- the west list command now accepts uncloned projects unless its
format string requires data that must be obtained from the project
repository.
- multi-repo commands now accept partial SHAs in more cases. For
example, "west init --mr SHA_PREFIX" works now.
The developer-visible additions to the west APIs are:
- west.log.banner(): new
- west.log.small_banner(): new
- west.manifest.Manifest.get_projects(): new
- west.manifest.Project.is_cloned(): new
- west.commands.WestCommand instances can now access the parsed
Manifest object via a new self.manifest property during the
do_run() call. If read, it returns the Manifest object or
aborts the command if it could not be parsed.
- west.manifest.Project.git() now has a capture_stderr kwarg
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
By default, skip fetches of revisions that resolve to SHAs or tag
names that are already available locally.
The user can override by using:
- west update -f always.
- a new update.fetch config option
Fixes: #274
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
The current way we handle errors when a command means an operation has
to be done on multiple projects (such as "west diff", "west update",
etc.) is unfortunate and a bit inconsistent.
In particular, "west update" tries in some cases to keep going and
update as many projects as possible when a rebase fails, but "west
diff" will abort on the first failure. Even worse, "west update" calls
the _git() helper in various places, which means that those commands
failing halts the entire update, leaving *only* failed rebases to
allow the process to continue.
Other commands, like "west forall", don't capture errors at all, so
the failure of a subcommand does not result in the overall process
failing. Whoops.
Let's clean up this mess by adding a helper function,
_handle_failed(), to our shared parent class, which makes sure that if
any error occurred, the whole command fails. Use it from diff, status,
update, and forall.
Change the command implementations as necessary so they never call
_git(), which allows us to roll up errors from anywhere in the command
procedure without aborting the program immediately. This lets us
inline quite a bit of code while we're here.
The lone exceptions in this file are init, list, and selfupdate.
The first two still fail immediately on the first error; the final one
continues to scream and die.
I think at this point I'm willing to say the manifest.py vs project.py
responsibilities are cleaned up:
- west.manifest.Project's git helpers are for general purpose and nondestructive
use.
- west.commands.project's use of Project and its git helpers are to do
destructive operations, but also invoking the nondestructive ones
- multi-project git operations have to be handled outside of
west.manifest, e.g. in _ProjectCommand
So:
Fixes: #191
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
The control flow when there is an unknown project is wrong: this is
calling is_cloned() on None. Indent it properly and add a regression
test.
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
Now that we have a bit more information on what happens when uncloned
projects are in the mix, handle the RuntimeError introduced in the
previous patch.
Clean up a stale comment and improve the help text while we're here.
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
These are also from code in project.py. It will be useful to have
similar behavior for extensions available without duplicating code, in
case the behavior changes.
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
Use self.manifest.get_projects() instead of the older implementation
of similar functionality. With the previous modification to west list
in place, no behavioral changes are expected.
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
That's unnecessary for almost all the format strings, and users may
want to get information about projects they don't have cloned in order
to decide whether or not to clone them.
Use the fact that we're delaying evaluation of all git related
commands to exit out only if the sha of an uncloned project is
requested. Otherwise, all the information we need is in the manifest.
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
This method allows finding a list of projects by name or path.
It's inspired by a helper method in projects.py, but has slightly
different semantics.
We'll move projects.py over to using it in the next patch, but having
this in manifest.py makes it API which is usable by extension commands.
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
This is a generally useful thing to know, and, like all the other
Project.git() wrappers, a nondestructive operation on the project
itself.
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
Just a cleanup we ought to do while we're refactoring in here.
Use util.canon_path(path), which is normcase(abspath(path)), instead
of normcase(realpath(path)) (in one case) or reimplementing the same
functionality (in a couple of others).
Regarding realpath:
- there's no difference on Windows (where abspath
and realpath are the same as symlinks are generally unsupported
there).
- on Unixes, avoiding realpath() means that symbolic links are not
derefenced, which is something we're trying to achieve in west in
general to support symlinks better.
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
Just a refactoring:
- remove exclude_manifest kwarg from _ProjectCommand._projects(),
using isinstance(..., ManifestProject) in the only user instead
- rename some variables
No functional changes expected, but this will enable moving the code
elsewhere, to make it available to extension commands which also want
similar functionality.
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
Now that we have a self.manifest attribute which returns the manifest
when it's available and fails gracefully when it isn't, we can avoid
re-parsing the manifest in the project command implementations.
Note this change is more convenient to make after having introduced
_ProjectCommand to allow accessing per-instance state from each of
these commands, instead of using standalone module-level functions.
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
This allows individual commands to avoid re-parsing. Properly
initialize it in the constructor (and topdir while we're here).
Use a property to get and set the manifest to allow us to fail with a
consistent error message when commands that require a manifest don't
get one.
We are basically saving the manifest similarly to topdir.
This opens up opportunities to avoid re-parsing the manifest
repeatedly.
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
I find the _add_parser() and _project() helper methods a bit
opaque.
- The parser boilerplate being avoided could be made easier to read by
being written in a more imperative style
- the project related helpers feel more like common instance
methods (since they need the WestCommand instance or knowledge about
its arguments to do their job) than true standalone functions.
Add a ProjectCommand superclass with helper methods for parsing and
doing some of the project related functionality instead of having
separate helper functions. This will pay dividends in subsequent
patches when we need to access more instance state.
We leave some other module-level functions, like _fetch(), where they
are for now to minimize changes.
Fix up tweak various pieces of help text while we're here -- there
were some really old mistakes left lying around from the 0.3 and 0.4
days. Yikes.
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
The diff and status commands with extra arguments never worked.
Remove the invalid test cases for them. The next patch is going to
fail this sort of invocation with an error instead of silently
ignoring it.
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
Rename _is_sha() to _maybe_sha() -- it's technically possible to have
commit refs which are branches that look like SHAs. Any branch name
that begins with a letter A-F an contains only hexadecimal characters
will work.
Allow revisions which are SHA prefixes. This makes it convenient to do
commands like "west init --mr some-sha-prefix" without the remote git
server barfing on an unknown ref.
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
The only caller of _fetch() is _update(). Move its definition to right
after _update()'s for readability. No functional changes expected.
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
It's not always possible to silence git commands with -q, and it is
sometimes necessary to do so to avoid spurious output being shown to
the user.
One example is using "git cat-file -e SOME_REV" to test if a revision
has been fetched and is available in the local object database. This
prints to stderr if it isn't. The command or other code running
Project.git() in this case may be able to detect and deal with the
missing revision appropriately without wanting the user to see any
messages, and that's currently not possible.
Fix it by adding a capture_stderr that works the same way
capture_stdout does, but for stderr.
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
Use memoization to avoid warning multiple times from Project.git() if
the git executable can't be found.
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
Add a simple option for validating the current manifest. Make sure to
print useful errors from this command whenever the manifest is
invalid, rather than dumping stack or being opaque about what went
wrong.
Mention this as a possibility for debugging in west --help when things
go wrong.
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
Document the possible exception behavior more clearly, and raise
FileNotFoundError if the manifest doesn't exist. This makes it
possible for users to detect this condition.
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
Save the SchemaError message in the MalformedManifest's string
representation if it is available. This makes it easier for users to
see what is wrong.
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
If the manifest can't be parsed, print a helpful message that the
extension commands can't be loaded.
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
We want to handle these ourselves. Without this patch, manifest parse
errors are printed in e.g. "west --version" output, which is not what
we want.
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
If the manifest is invalid, we can still run some built-in
commands (like west config), or print help output.
Don't spit up an exception when we can't find extension commands to
allow built-ins to continue. Catch MalformedManifest and
MalformedConfig at top level to handle anything that slips through
without dumping stack.
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
Compared to rc1, this adds a bugfix and a new feature, the project
'repo-path' attribute, to allow continuing use of remotes with
non-unique project names.
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
Users have provided negative feedback about some specifics related to
the new restriction that project names must be unique.
In particular, there is a use case for fetching the same project (the
LittleVGL graphics library) into two separate paths with two separate
revisions (to allow Kconfig to select between two major versions).
Since project names must now be unique, this would require the use of
the `url` attribute instead of `remote` (since the URL is of course
the same for both "projects", "remote.url-base + name" cannot be used
without violating name uniqueness).
To make it possible to keep using remotes, add a new optional
'repo-path' attribute to each project. If given, then 'remote.url-base
+ repo-path' forms the fetch URL.
Like remote itself, 'repo-path' cannot be combined with 'url'.
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
Manifests which use both default remotes and projects with URL
elements may incorrectly be rejected as invalid. Fix this and add a
regression test.
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
This is needed for find_namespace_packages. Make sure CI upgrades it
as well; the default shippable container doesn't meet the requirement.
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
Marti Bolivar