We're seeing warnings when people have ZEPHYR_BASE set to the
"<directory>" they pass to "west init -l <directory>".
Avoid them exactly in the situation that <directory> is a zephyr
repository by using the manifest directory as ZEPHYR_BASE. For
upstream, that is correct anyway.
Merge a couple of lines while here as a cosmetic improvement.
Fixes: #167
Signed-off-by: Marti Bolivar <marti@foundries.io>
This commit cleans up the second fetch, which potentially could have
changed content since first fetch.
Normal ref-space (similar to git clone) and user specified revision is
now fetched in same command.
This further allows to checkout tags in detached HEAD while still
referring to the tag name instead of FETCH_HEAD.
This commit also allows a user to specify a sha.
When specify a sha as revision, that sha must be valid in to the normal
ref-space.
Signed-off-by: Torsten Rasmussen <torsten.rasmussen@nordicsemi.no>
Changing `west init` into using
- git init
- git fetch
- git checkout
as this allows users to specify other refs, such as pull/<no>/head
which is not possible with git clone on github.
Signed-off-by: Torsten Rasmussen <torsten.rasmussen@nordicsemi.no>
For various reasons, we don't load extension command modules until the
user has requested that we run a particular extension command.
As is, this limits the "west -h" output to the extension command names
and the projects which define them, which isn't as nice as what
built-in commands get.
Allow extension commands to provide help text by extending the schema
for each command entry with an optional "help" key, and display it in
the output. If it is missing, a default is provided which recommends
just running "west $EXTENSION_COMMAND_NAME -h".
Fix the help formatting by adding a colon after each extension command
name to match the style used for built-ins while we are here.
Keep test cases up to date.
Signed-off-by: Marti Bolivar <marti@foundries.io>
Cut release 0.5.0. This is the initial tree provided to the wider
Zephyr developer community during the Zephyr v1.14 development cycle.
Signed-off-by: Marti Bolivar <marti@foundries.io>
This commit fixes:
- Issue if multiple projects defines the same command.
This will print a warning for project/command not being available
- Fixed issue where same command could be listed multiple times
Signed-off-by: Torsten Rasmussen <torsten.rasmussen@nordicsemi.no>
Signed-off-by: Marti Bolivar <marti@foundries.io>
The current output looks kind of silly. Tweak it and add a TODO
comment for further noncritical improvements.
Signed-off-by: Marti Bolivar <marti@foundries.io>
Add the python file's directory to sys.path, to make it easier to
import other modules in the same directory.
Signed-off-by: Marti Bolivar <marti@foundries.io>
These are moving back to the zephyr repository as west extension
commands. This will (finally) let us remove the copy of west inside
zephyr to make those features available, and lets us keep a separate
west repository for bootstrapping and multirepo management while still
making it easier for people to just edit their runners.
Since this just leaves project and hidden commands, we don't print
indented groups of built-in commands, as that would be ugly and not
make sense.
Signed-off-by: Marti Bolivar <marti@foundries.io>
Added tests to ensure missing extension files (in case a project in
not cloned) will not cause west built-in commands to fail.
Signed-off-by: Torsten Rasmussen <torsten.rasmussen@nordicsemi.no>
On windows a couple of manifest test cases were failing due to path
delimiter issues.
This commit fixes those test cases by using os.path.sep instead of '/'
Signed-off-by: Torsten Rasmussen <torsten.rasmussen@nordicsemi.no>
This commit creates additional tests for 'west init -l' to ensure
behaviour when initializing west around an existing manifest
repository.
Signed-off-by: Torsten Rasmussen <torsten.rasmussen@nordicsemi.no>
Ability to initialize west in a folder which contains a manifest
project (as example by an earlier 'git clone') but which is in west
context uninitialized.
Signed-off-by: Torsten Rasmussen <torsten.rasmussen@nordicsemi.no>
Removed url and revision from west config file from west config file as
they are no longer needed when manifest repository can be handled
directly with git
Signed-off-by: Torsten Rasmussen <torsten.rasmussen@nordicsemi.no>
New update command will clone, fetch, and checkout a detached HEAD
according to manifest file.
Signed-off-by: Torsten Rasmussen <torsten.rasmussen@nordicsemi.no>
Renamed 'west update' to 'west selfupdate' to keep it separated from
future update command.
Signed-off-by: Torsten Rasmussen <torsten.rasmussen@nordicsemi.no>
Add logic to retrieve external command specifications grouped by the
project which provides them. Filter these so built-in commands never
get shadowed, and wire them up to the argument parsing and help
printing logic. Add a special-case command handler for these which
re-parses command line arguments after instantiating the command
object and installing its parser.
Signed-off-by: Marti Bolivar <marti@foundries.io>
Add support for a new optional 'west-commands' key in each project in
a manifest. This specifies the location of a file which describes
additional west commands provided by that project.
Extend the Manifest class to detect this key and record the
information inside. Nothing else is done about external commands in
that class, since its purpose is just to parse the manifest.
Extend west.commands with a new external_commands() function, which
consumes a Manifest and returns a list of "descriptors" of external
commands the manifest's projects declare. These descriptors contain
metadata about the commands, but do not actually import any Python
modules related to them or otherwise try to instantiate the relevant
WestCommand instances.
This is for two reasons:
1. Performance: don't import what you might not need
2. Security: don't import random code you downloaded somewhere on the
internet! Importing a module you got from somewhere random is
basically curl | sh.
This infrastructure will be wired into main() in a subsequent patch.
Signed-off-by: Marti Bolivar <marti@foundries.io>
Some utilities which use subcommands break them into groups in their
help output, to make it easier for the reader to find what they are
looking for. Two notable examples are docker and git.
Do the same thing for west, by tweaking the COMMANDS type to group
related commands together under a string key which describes the
group, and adding a custom argument parser subclass which uses this
information when formatting the top-level command help.
Add a group of "hidden" commands which is just PostInit for
now. Eventually that will be going away, and in the meantime, we'd
like to sweep it under the rug.
I would have preferred to do this by hooking into argparse itself, but
it is annoyingly opaque about what constitutes stable API.
The top-level "west -h" output now looks like this (without ">"s):
> $ west -h
> usage: west [-h] [-z ZEPHYR_BASE] [-v] [-V] <command> ...
>
> The Zephyr RTOS meta-tool.
>
> optional arguments:
> -h, --help show this help message and exit
> -z ZEPHYR_BASE, --zephyr-base ZEPHYR_BASE
> Override the Zephyr base directory. The
> default is the manifest project with path
> "zephyr".
> -v, --verbose Display verbose output. May be given multiple
> times to increase verbosity.
> -V, --version print the program version and exit
>
> west commands used in various situations:
> building and running zephyr:
> build: compile a Zephyr application
> flash: flash and run a binary on a board
> debug: flash and interactively debug a Zephyr
> application
> debugserver: connect to board and launch a debug server
> attach: interactively debug a board
>
> managing multiple repositories in the installation:
> list: print information about projects in the west
> manifest
> clone: clone remote projects into local installation
> fetch: "git fetch" changes from project remotes
> pull: "git pull" changes from project remotes
> rebase: "git rebase" local projects onto manifest
> versions
> branch: create a branch in one or more local projects
> checkout: check out a branch in one or more local
> projects
> diff: "git diff" for one or more projects
> status: "git status" for one or more projects
> update: update the manifest and west repositories
> forall: run a command in one or more local projects
>
> Run "west <command> -h" for help on each command.
The command-level help (e.g. "west build -h") is not changed.
Fixes: #107
Signed-off-by: Marti Bolivar <marti@foundries.io>
This is a baby step on issue #107 (Improve `west --help` formatting).
Add per-command help output to each command. While we're here, rename
'init' to 'post-init' to avoid a name clash with the init command
handled by the bootstrapper.
The "commands:" help output now looks like this in west -h:
positional arguments:
<command>
build compile a Zephyr application
flash flash and run a binary on a board
debug flash and interactively debug a Zephyr application
debugserver connect to board and launch a debug server
attach interactively debug a board
post-init finish init tasks
list print information about projects in the west manifest
clone clone remote projects into local installation
fetch "git fetch" changes from project remotes
pull "git pull" changes from project remotes
rebase "git rebase" local projects onto manifest versions
branch create a branch in one or more local projects
checkout check out a branch in one or more local projects
diff "git diff" for one or more projects
status "git status" for one or more projects
update update the manifest and west repositories
forall run a command in one or more local projects
It would be nice to be able to group these subparsers and hide the
post-init command, though, which we can't seem to do with plain
argparse.
Signed-off-by: Marti Bolivar <marti@foundries.io>
In order to add support for extension commands, we will need to know
where the manifest is before we parse any command line arguments.
This means the -m argument supported by the existing project commands
has to go away. We'll always let the manifest module fetch the
contents from the location specified in the configuration file.
(The Manifest class is still capable of parsing manifests from
arbitrary file system paths.)
While we're here, add a validity check for the sections parameter to
the Manifest constructor and factory methods. This wasn't being used
properly: it should be a list. The fact that strings are sequences
means it basically gets silently ignored when passing a string.
Signed-off-by: Marti Bolivar <marti@foundries.io>
Let's not deal with META_NAMES anymore, which is only used once. We
can emit a nicer error message for the user without it.
Signed-off-by: Marti Bolivar <marti@foundries.io>
- Fix comments and project help messages that refer to the manifest
as a distinct special repository
- Remove extra code that assumes multiple meta projects may exist
Signed-off-by: Marti Bolivar <marti@foundries.io>
Calling bootstrap() from the WestNotFound exception handler in init()
is making the WestNotFound show up in stack traces for errors in
bootstrap(), which makes for confusing output. Fix that by moving the
call to bootstrap() out.
Signed-off-by: Marti Bolivar <marti@foundries.io>
A git clone of manifest repository into temporary folder on windows
will contain read-only files.
This will cause shutil.rmtree to fail and abort bootstrapping.
Current fix is to ignore such errors on windows and rely on windows
temp folder cleanup.
Signed-off-by: Torsten Rasmussen <torsten.rasmussen@nordicsemi.no>
For better handling of errors the Manifest class now raises a
MalformedConfig exception if required settings are missing in the west
config file
Signed-off-by: Torsten Rasmussen <torsten.rasmussen@nordicsemi.no>
Topdir marker '.west_topdir' removed.
West now determines the topdir by looking for the '.west' folder.
Signed-off-by: Torsten Rasmussen <torsten.rasmussen@nordicsemi.no>
Added west init command to west.main to support post-process of
west init.
Added --use-cache to limit network use when cloning manifest project
repository
Signed-off-by: Torsten Rasmussen <torsten.rasmussen@nordicsemi.no>
- self key introduced in manifest file to support local path location
for the repository containing the manifest file
- Removed manifest repository under west folder
Signed-off-by: Torsten Rasmussen <torsten.rasmussen@nordicsemi.no>
As part of moving manifest file inside a project repository, the west
folder is renamed to .west as it will no longer contain a manifest
repository (and potentially also not west repo in future)
Signed-off-by: Torsten Rasmussen <torsten.rasmussen@nordicsemi.no>
As part of moving manifest file inside a project reposity, the manifest
file is renamed from default.yml to west.yml
Signed-off-by: Torsten Rasmussen <torsten.rasmussen@nordicsemi.no>
Add a flake8 configuration to the tox testing. This enforces a
consistent style. Clean up remaining issues around the codebase.
Disable a handful of flake8's complaints when the style seems popular
enough among the team members to ignore.
Signed-off-by: Marti Bolivar <marti@foundries.io>
To properly test the project commands, it would be best to have a
fresh west bootstrapper package created and installed on PATH, so it
could be used to run commands exactly as they'd happen if we package
and ship the working tree.
To make that easier, add a dependency on tox and use it for testing:
https://tox.readthedocs.io/en/latest/
From now on, we'll test west by running 'tox' from the repository
root. This has several advantages over running pytest directly:
- "Just run tox": there are no longer any differences in test invocation
between POSIX OSes and Windows.
- tox creates an sdist package of the current tree using our setup.py
and installs it into a new virtual environment, then runs tests
there. This removes interference from other packages installed on
the host (like released bootstrappers that are also installed)
- we get to run multiple shell commands in order, should that ever be needed,
in our test procedures in a way that won't affect users
With that done, we can re-work the multirepo command testing to invoke
the bootstrapper in the virtual environment, adding various tests and
filling in longstanding testing gaps by adding increased checking of
the results (currently, much of the testing just checks whether
commands do or do not error out, which isn't enough).
These changes were made with a view towards the upcoming changes which
are planned before releasing west "into the wild": the test case code
should be mostly the same before and after the changes, so this serves
as a good baseline against regressions introduced by those upcoming
changes.
Signed-off-by: Marti Bolivar <marti@foundries.io>
[wip] debugging shippable results
Signed-off-by: Marti Bolivar <marti@foundries.io>
[wip] just test one py3
shutil.which west is picking up a 3.4 version in the 3.6 test, oddly
Signed-off-by: Marti Bolivar <marti@foundries.io>
This prevents random wests in developer PYTHONPATHs from coming
first. It's being done for convenience during development and testing.
Signed-off-by: Marti Bolivar <marti@foundries.io>
Git clone can take a raw file system path instead of a URL when
cloning a repository that is already available locally somewhere else.
When this happens, git tries to use hard links to create the files in
the destination, which is more efficient, so the limitation to URLs is
not good. Remove it.
Signed-off-by: Marti Bolivar <marti@foundries.io>
The _expand_shorthands() helper function allows constructing strings
based on attributes of a project. However, it uses a custom format
string DSL-like syntax instead of picking one of the two which already
exist in Python.
Convert this to use the standard .format() string formatter.
This is mostly a mechanical change. The only thing worth calling out
is that when using .format() to construct something that will get
passed to _expand_shorthands(), we need to double the brackets for the
shorthand keys (so "{{" / "}}" instead of "{" / "}") so that the first
.format interprets them as ordinary brackets, which the call to
.format() in _expand_shorthands() then sees as an access to a format
value by name.
Signed-off-by: Marti Bolivar <marti@foundries.io>
Miscellaneous fixes and improvements to the manifest parsing code.
- remove unused and undocumented 'url' parameter to Project
constructor: this only matters for SpecialProject instances
- add and fix pydoc parameter documentation for a few classes
- following manifest schema changes, rename the Remote 'url' parameter
to 'url_base'
- NotImplemented is not an exception; it should be returned, not
raised
- pass project paths through os.path.normpath() when initializing
Project instances
Signed-off-by: Marti Bolivar <marti@foundries.io>
Marti Bolivar