Ability to initialize west in a folder which contains a manifest
project (as example by an earlier 'git clone') but which is in west
context uninitialized.
Signed-off-by: Torsten Rasmussen <torsten.rasmussen@nordicsemi.no>
Removed url and revision from west config file from west config file as
they are no longer needed when manifest repository can be handled
directly with git
Signed-off-by: Torsten Rasmussen <torsten.rasmussen@nordicsemi.no>
New update command will clone, fetch, and checkout a detached HEAD
according to manifest file.
Signed-off-by: Torsten Rasmussen <torsten.rasmussen@nordicsemi.no>
Renamed 'west update' to 'west selfupdate' to keep it separated from
future update command.
Signed-off-by: Torsten Rasmussen <torsten.rasmussen@nordicsemi.no>
Add logic to retrieve external command specifications grouped by the
project which provides them. Filter these so built-in commands never
get shadowed, and wire them up to the argument parsing and help
printing logic. Add a special-case command handler for these which
re-parses command line arguments after instantiating the command
object and installing its parser.
Signed-off-by: Marti Bolivar <marti@foundries.io>
Add support for a new optional 'west-commands' key in each project in
a manifest. This specifies the location of a file which describes
additional west commands provided by that project.
Extend the Manifest class to detect this key and record the
information inside. Nothing else is done about external commands in
that class, since its purpose is just to parse the manifest.
Extend west.commands with a new external_commands() function, which
consumes a Manifest and returns a list of "descriptors" of external
commands the manifest's projects declare. These descriptors contain
metadata about the commands, but do not actually import any Python
modules related to them or otherwise try to instantiate the relevant
WestCommand instances.
This is for two reasons:
1. Performance: don't import what you might not need
2. Security: don't import random code you downloaded somewhere on the
internet! Importing a module you got from somewhere random is
basically curl | sh.
This infrastructure will be wired into main() in a subsequent patch.
Signed-off-by: Marti Bolivar <marti@foundries.io>
Some utilities which use subcommands break them into groups in their
help output, to make it easier for the reader to find what they are
looking for. Two notable examples are docker and git.
Do the same thing for west, by tweaking the COMMANDS type to group
related commands together under a string key which describes the
group, and adding a custom argument parser subclass which uses this
information when formatting the top-level command help.
Add a group of "hidden" commands which is just PostInit for
now. Eventually that will be going away, and in the meantime, we'd
like to sweep it under the rug.
I would have preferred to do this by hooking into argparse itself, but
it is annoyingly opaque about what constitutes stable API.
The top-level "west -h" output now looks like this (without ">"s):
> $ west -h
> usage: west [-h] [-z ZEPHYR_BASE] [-v] [-V] <command> ...
>
> The Zephyr RTOS meta-tool.
>
> optional arguments:
> -h, --help show this help message and exit
> -z ZEPHYR_BASE, --zephyr-base ZEPHYR_BASE
> Override the Zephyr base directory. The
> default is the manifest project with path
> "zephyr".
> -v, --verbose Display verbose output. May be given multiple
> times to increase verbosity.
> -V, --version print the program version and exit
>
> west commands used in various situations:
> building and running zephyr:
> build: compile a Zephyr application
> flash: flash and run a binary on a board
> debug: flash and interactively debug a Zephyr
> application
> debugserver: connect to board and launch a debug server
> attach: interactively debug a board
>
> managing multiple repositories in the installation:
> list: print information about projects in the west
> manifest
> clone: clone remote projects into local installation
> fetch: "git fetch" changes from project remotes
> pull: "git pull" changes from project remotes
> rebase: "git rebase" local projects onto manifest
> versions
> branch: create a branch in one or more local projects
> checkout: check out a branch in one or more local
> projects
> diff: "git diff" for one or more projects
> status: "git status" for one or more projects
> update: update the manifest and west repositories
> forall: run a command in one or more local projects
>
> Run "west <command> -h" for help on each command.
The command-level help (e.g. "west build -h") is not changed.
Fixes: #107
Signed-off-by: Marti Bolivar <marti@foundries.io>
This is a baby step on issue #107 (Improve `west --help` formatting).
Add per-command help output to each command. While we're here, rename
'init' to 'post-init' to avoid a name clash with the init command
handled by the bootstrapper.
The "commands:" help output now looks like this in west -h:
positional arguments:
<command>
build compile a Zephyr application
flash flash and run a binary on a board
debug flash and interactively debug a Zephyr application
debugserver connect to board and launch a debug server
attach interactively debug a board
post-init finish init tasks
list print information about projects in the west manifest
clone clone remote projects into local installation
fetch "git fetch" changes from project remotes
pull "git pull" changes from project remotes
rebase "git rebase" local projects onto manifest versions
branch create a branch in one or more local projects
checkout check out a branch in one or more local projects
diff "git diff" for one or more projects
status "git status" for one or more projects
update update the manifest and west repositories
forall run a command in one or more local projects
It would be nice to be able to group these subparsers and hide the
post-init command, though, which we can't seem to do with plain
argparse.
Signed-off-by: Marti Bolivar <marti@foundries.io>
In order to add support for extension commands, we will need to know
where the manifest is before we parse any command line arguments.
This means the -m argument supported by the existing project commands
has to go away. We'll always let the manifest module fetch the
contents from the location specified in the configuration file.
(The Manifest class is still capable of parsing manifests from
arbitrary file system paths.)
While we're here, add a validity check for the sections parameter to
the Manifest constructor and factory methods. This wasn't being used
properly: it should be a list. The fact that strings are sequences
means it basically gets silently ignored when passing a string.
Signed-off-by: Marti Bolivar <marti@foundries.io>
Let's not deal with META_NAMES anymore, which is only used once. We
can emit a nicer error message for the user without it.
Signed-off-by: Marti Bolivar <marti@foundries.io>
- Fix comments and project help messages that refer to the manifest
as a distinct special repository
- Remove extra code that assumes multiple meta projects may exist
Signed-off-by: Marti Bolivar <marti@foundries.io>
Calling bootstrap() from the WestNotFound exception handler in init()
is making the WestNotFound show up in stack traces for errors in
bootstrap(), which makes for confusing output. Fix that by moving the
call to bootstrap() out.
Signed-off-by: Marti Bolivar <marti@foundries.io>
A git clone of manifest repository into temporary folder on windows
will contain read-only files.
This will cause shutil.rmtree to fail and abort bootstrapping.
Current fix is to ignore such errors on windows and rely on windows
temp folder cleanup.
Signed-off-by: Torsten Rasmussen <torsten.rasmussen@nordicsemi.no>
For better handling of errors the Manifest class now raises a
MalformedConfig exception if required settings are missing in the west
config file
Signed-off-by: Torsten Rasmussen <torsten.rasmussen@nordicsemi.no>
Topdir marker '.west_topdir' removed.
West now determines the topdir by looking for the '.west' folder.
Signed-off-by: Torsten Rasmussen <torsten.rasmussen@nordicsemi.no>
Added west init command to west.main to support post-process of
west init.
Added --use-cache to limit network use when cloning manifest project
repository
Signed-off-by: Torsten Rasmussen <torsten.rasmussen@nordicsemi.no>
- self key introduced in manifest file to support local path location
for the repository containing the manifest file
- Removed manifest repository under west folder
Signed-off-by: Torsten Rasmussen <torsten.rasmussen@nordicsemi.no>
As part of moving manifest file inside a project repository, the west
folder is renamed to .west as it will no longer contain a manifest
repository (and potentially also not west repo in future)
Signed-off-by: Torsten Rasmussen <torsten.rasmussen@nordicsemi.no>
As part of moving manifest file inside a project reposity, the manifest
file is renamed from default.yml to west.yml
Signed-off-by: Torsten Rasmussen <torsten.rasmussen@nordicsemi.no>
Add a flake8 configuration to the tox testing. This enforces a
consistent style. Clean up remaining issues around the codebase.
Disable a handful of flake8's complaints when the style seems popular
enough among the team members to ignore.
Signed-off-by: Marti Bolivar <marti@foundries.io>
To properly test the project commands, it would be best to have a
fresh west bootstrapper package created and installed on PATH, so it
could be used to run commands exactly as they'd happen if we package
and ship the working tree.
To make that easier, add a dependency on tox and use it for testing:
https://tox.readthedocs.io/en/latest/
From now on, we'll test west by running 'tox' from the repository
root. This has several advantages over running pytest directly:
- "Just run tox": there are no longer any differences in test invocation
between POSIX OSes and Windows.
- tox creates an sdist package of the current tree using our setup.py
and installs it into a new virtual environment, then runs tests
there. This removes interference from other packages installed on
the host (like released bootstrappers that are also installed)
- we get to run multiple shell commands in order, should that ever be needed,
in our test procedures in a way that won't affect users
With that done, we can re-work the multirepo command testing to invoke
the bootstrapper in the virtual environment, adding various tests and
filling in longstanding testing gaps by adding increased checking of
the results (currently, much of the testing just checks whether
commands do or do not error out, which isn't enough).
These changes were made with a view towards the upcoming changes which
are planned before releasing west "into the wild": the test case code
should be mostly the same before and after the changes, so this serves
as a good baseline against regressions introduced by those upcoming
changes.
Signed-off-by: Marti Bolivar <marti@foundries.io>
[wip] debugging shippable results
Signed-off-by: Marti Bolivar <marti@foundries.io>
[wip] just test one py3
shutil.which west is picking up a 3.4 version in the 3.6 test, oddly
Signed-off-by: Marti Bolivar <marti@foundries.io>
This prevents random wests in developer PYTHONPATHs from coming
first. It's being done for convenience during development and testing.
Signed-off-by: Marti Bolivar <marti@foundries.io>
Git clone can take a raw file system path instead of a URL when
cloning a repository that is already available locally somewhere else.
When this happens, git tries to use hard links to create the files in
the destination, which is more efficient, so the limitation to URLs is
not good. Remove it.
Signed-off-by: Marti Bolivar <marti@foundries.io>
The _expand_shorthands() helper function allows constructing strings
based on attributes of a project. However, it uses a custom format
string DSL-like syntax instead of picking one of the two which already
exist in Python.
Convert this to use the standard .format() string formatter.
This is mostly a mechanical change. The only thing worth calling out
is that when using .format() to construct something that will get
passed to _expand_shorthands(), we need to double the brackets for the
shorthand keys (so "{{" / "}}" instead of "{" / "}") so that the first
.format interprets them as ordinary brackets, which the call to
.format() in _expand_shorthands() then sees as an access to a format
value by name.
Signed-off-by: Marti Bolivar <marti@foundries.io>
Miscellaneous fixes and improvements to the manifest parsing code.
- remove unused and undocumented 'url' parameter to Project
constructor: this only matters for SpecialProject instances
- add and fix pydoc parameter documentation for a few classes
- following manifest schema changes, rename the Remote 'url' parameter
to 'url_base'
- NotImplemented is not an exception; it should be returned, not
raised
- pass project paths through os.path.normpath() when initializing
Project instances
Signed-off-by: Marti Bolivar <marti@foundries.io>
We need to do another copy of west into Zephyr to get some runner
fixes and features, but some assumptions have crept back into the code
that we're in a multirepo install.
Fix these so we can use flash and debug commands in a monorepo Zephyr
installation.
Signed-off-by: Marti Bolivar <marti@foundries.io>
On Microsoft Windows, the default name of the JLink executable is
"JLink.exe", while on *NIX it is instead "JLinkExe". Detect and
correctly set the default name based on the platform.
Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
Fixes: #139
Fixes the issue of fetching SHAs when specified in manifest file.
It is checked if a revision is a SHA, in which case the fetch is
performed on the url and afterwards the manifest-rev reference is
updated to the SHA.
Signed-off-by: Torsten Rasmussen <torsten.rasmussen@nordicsemi.no>
Fixing the behavior of 'west debug' requires users to update their
bootstrap installation. Bump the minor release number to make that
possible.
Signed-off-by: Marti Bolivar <marti@foundries.io>
Commit f8bd521 ("Add self-update for West and the manifest") switched
from importing the west module and invoking main directly from the
same process as the bootstrapper to invoking it as a subprocess.
This breaks 'west debug'.
Runners which run GDB directly to debug a target need to be able to
ignore SIGINT while GDB is running, as Ctrl-C halts the target and
returns control to the (gdb) prompt instead of terminating the program
as usual.
Because of that, the runner code must be running in the same process
as the bootstrapper, or the parent (bootstrap) process, which doesn't
ignore the signal, means the control C at the shell gets delivered to
a process which uses the default handler to exit the program.
Fix it by going back to the mechanism the bootstrapper originally
used to delegate a command to the installed west.
Fixes: #135
Signed-off-by: Marti Bolivar <marti@foundries.io>
The argument array to main is in the argv argument, *not* sys.argv.
Fix this.
(Using sys.argv directly prevents the main function from getting its
arguments from an arbitrary value specified by the caller.)
Signed-off-by: Marti Bolivar <marti@foundries.io>
It's not clear from the help how the source and build directories are
determined. Fix that and sentence-case the other help text while we're
here.
Signed-off-by: Marti Bolivar <marti@foundries.io>
Fixes: #119
This commit changes the way show-ref is executed in order to determine
whether HEAD is pointing to a valid reference or if the repository is
in initialized stated.
Signed-off-by: Torsten Rasmussen <torsten.rasmussen@nordicsemi.no>
This commit uses url-base instead of url to avoid confusion for users
as url in west section of manifest is full url, where url-base in manifest
section will be concatenated with project name to for clone url.
Signed-off-by: Torsten Rasmussen <torsten.rasmussen@nordicsemi.no>
Torsten Rasmussen