It is always possible to satisfy this constraint now that project URLs
may be specified explicitly.
(This restriction will be necessary to make manifest imports work
sensibly -- we should have done this before...).
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
We're going to add explicit project URLs to the manifest, so the
remote should no longer be a positional.
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
It takes 20 seconds on my machine to run the full set of tests, which
is slow enough that testing breaks me out of flow state.
On the suspicion that creating git repositories and using the file://
protocol when cloning (which prevents use of hardlinks) is slowing
things down, use some pytest features to avoid creating git
repositories repeatedly. Also let git use hardlinks when they are
available when cloning repositories.
On my system, this brings the average of 10 runs from 20.129 seconds
spent testing to 17.649, a 12% improvement overall. Still not ideal,
but not worth throwing away, either.
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
Don't copy the west tree; tox already installs it for us into the new
virtualenv, and we don't run any code out of a checked out repository
anymore, so doing things related to that is unnecessary.
This also makes the tests run a little bit faster (around a 5% or more
speedup on my system).
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
It's not clear what this is doing here. It doesn't begin with "test",
so it's not being tested, and it seems to be a copy of test_config.py
with some important features needed to avoid modifying the user's
configuration.
Delete it.
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
There's no good reason to have these tests in subdirectories. Flatten
them out. Keep the directory of invalid manifests separate to keep the
directory listing clean, though.
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
Get rid of the separate bootstrapper. Implement 'west init' as a
regular WestCommand which combines the bootstrapper + PostInit.
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
Fix or improve various docstrings so they are usable from Sphinx. Without this
patch, we can't provide API documentation for west from the zephyr docs tree.
Remove comments from the various schema files. We'll move this information into
the zephyr documentation.
Use "extension" instead of "external" in some log messages for consistency with
the final name of the pluggable commands that aren't built-in.
No behavioral changes other than the slightly different printlines.
Fixes: #240
Signed-off-by: Marti Bolivar <marti@foundries.io>
It has been discovered that in the event that a different remote is
added to manifest file which contains same branches but cannot be
fast forwarded into the local refs space, then a west update will fail
with the error:
! [rejected] <branch> -> refs/west/<branch> (non-fast-forward)
This commit fixes this issue by introducing '-f' when fetching.
This commit has additional test for verifying the fix.
Signed-off-by: Torsten Rasmussen <torsten.rasmussen@nordicsemi.no>
Fixes: #198#213
This commit add the command `west config` for getting and setting of
key - value pairs in:
- Project specific: `<project>/.west/config`
- Global specific: `~/.westconfig`
- System specific, Linux:`/etc/westconfig`
MacOS: `/usr/local/etc/westconfig`
Windows: `%PROGRAMDATA%/west/config`
It also includes corresponding test cases.
Signed-off-by: Torsten Rasmussen <torsten.rasmussen@nordicsemi.no>
As part of introducing 'west config' command, the current config.py is
renamed into configuration.py in order to have distinct name for each
module as west commands have module named same as command.
Signed-off-by: Torsten Rasmussen <torsten.rasmussen@nordicsemi.no>
A new test compares the output of -h with that of help for all
commands.
In order to be able to test commands other than the project ones,
refactor the helper functions and fixtures into a common conftest.py and
rename the enclosing folder from "project" to "commands".
Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
Since the python object is named url_base, this field is getting
frozen via as_dict with an underscore rather than the dash required
by the schema.
Signed-off-by: Andy Doan <andy@foundries.io>
To align the way `west selfupdate` and `west update` works then
`west selfupdate` has now been adjusted to support --rebase and
--keep-descendants to align functionality with `west update`.
This should make it easier to describe and understand the overall
update functionality in west.
Signed-off-by: Torsten Rasmussen <torsten.rasmussen@nordicsemi.no>
The behavior of 'west list SOME_INVALID_PROJECT' (and any other
commands which try to match SOME_INVALID_PROJECT with real project)
is incorrect when called from a project directory: the current
project is found instead of an error being returned.
Fix this by requiring that, when treated as a path, the given argument
exactly normalizes to a project directory. Otherwise, it is possible
for it to match nonexistent files inside of the existing project
directory.
Add some more testing. The ones that expect a CalledProcessError fail
without this project.py patch. With it, they all pass.
Signed-off-by: Marti Bolivar <marti@foundries.io>
For various reasons, we don't load extension command modules until the
user has requested that we run a particular extension command.
As is, this limits the "west -h" output to the extension command names
and the projects which define them, which isn't as nice as what
built-in commands get.
Allow extension commands to provide help text by extending the schema
for each command entry with an optional "help" key, and display it in
the output. If it is missing, a default is provided which recommends
just running "west $EXTENSION_COMMAND_NAME -h".
Fix the help formatting by adding a colon after each extension command
name to match the style used for built-ins while we are here.
Keep test cases up to date.
Signed-off-by: Marti Bolivar <marti@foundries.io>
This commit fixes:
- Issue if multiple projects defines the same command.
This will print a warning for project/command not being available
- Fixed issue where same command could be listed multiple times
Signed-off-by: Torsten Rasmussen <torsten.rasmussen@nordicsemi.no>
Signed-off-by: Marti Bolivar <marti@foundries.io>
These are moving back to the zephyr repository as west extension
commands. This will (finally) let us remove the copy of west inside
zephyr to make those features available, and lets us keep a separate
west repository for bootstrapping and multirepo management while still
making it easier for people to just edit their runners.
Since this just leaves project and hidden commands, we don't print
indented groups of built-in commands, as that would be ugly and not
make sense.
Signed-off-by: Marti Bolivar <marti@foundries.io>
Added tests to ensure missing extension files (in case a project in
not cloned) will not cause west built-in commands to fail.
Signed-off-by: Torsten Rasmussen <torsten.rasmussen@nordicsemi.no>
On windows a couple of manifest test cases were failing due to path
delimiter issues.
This commit fixes those test cases by using os.path.sep instead of '/'
Signed-off-by: Torsten Rasmussen <torsten.rasmussen@nordicsemi.no>
This commit creates additional tests for 'west init -l' to ensure
behaviour when initializing west around an existing manifest
repository.
Signed-off-by: Torsten Rasmussen <torsten.rasmussen@nordicsemi.no>
Removed url and revision from west config file from west config file as
they are no longer needed when manifest repository can be handled
directly with git
Signed-off-by: Torsten Rasmussen <torsten.rasmussen@nordicsemi.no>
New update command will clone, fetch, and checkout a detached HEAD
according to manifest file.
Signed-off-by: Torsten Rasmussen <torsten.rasmussen@nordicsemi.no>
Renamed 'west update' to 'west selfupdate' to keep it separated from
future update command.
Signed-off-by: Torsten Rasmussen <torsten.rasmussen@nordicsemi.no>
Add support for a new optional 'west-commands' key in each project in
a manifest. This specifies the location of a file which describes
additional west commands provided by that project.
Extend the Manifest class to detect this key and record the
information inside. Nothing else is done about external commands in
that class, since its purpose is just to parse the manifest.
Extend west.commands with a new external_commands() function, which
consumes a Manifest and returns a list of "descriptors" of external
commands the manifest's projects declare. These descriptors contain
metadata about the commands, but do not actually import any Python
modules related to them or otherwise try to instantiate the relevant
WestCommand instances.
This is for two reasons:
1. Performance: don't import what you might not need
2. Security: don't import random code you downloaded somewhere on the
internet! Importing a module you got from somewhere random is
basically curl | sh.
This infrastructure will be wired into main() in a subsequent patch.
Signed-off-by: Marti Bolivar <marti@foundries.io>
In order to add support for extension commands, we will need to know
where the manifest is before we parse any command line arguments.
This means the -m argument supported by the existing project commands
has to go away. We'll always let the manifest module fetch the
contents from the location specified in the configuration file.
(The Manifest class is still capable of parsing manifests from
arbitrary file system paths.)
While we're here, add a validity check for the sections parameter to
the Manifest constructor and factory methods. This wasn't being used
properly: it should be a list. The fact that strings are sequences
means it basically gets silently ignored when passing a string.
Signed-off-by: Marti Bolivar <marti@foundries.io>
Added west init command to west.main to support post-process of
west init.
Added --use-cache to limit network use when cloning manifest project
repository
Signed-off-by: Torsten Rasmussen <torsten.rasmussen@nordicsemi.no>
- self key introduced in manifest file to support local path location
for the repository containing the manifest file
- Removed manifest repository under west folder
Signed-off-by: Torsten Rasmussen <torsten.rasmussen@nordicsemi.no>
As part of moving manifest file inside a project repository, the west
folder is renamed to .west as it will no longer contain a manifest
repository (and potentially also not west repo in future)
Signed-off-by: Torsten Rasmussen <torsten.rasmussen@nordicsemi.no>
As part of moving manifest file inside a project reposity, the manifest
file is renamed from default.yml to west.yml
Signed-off-by: Torsten Rasmussen <torsten.rasmussen@nordicsemi.no>
Add a flake8 configuration to the tox testing. This enforces a
consistent style. Clean up remaining issues around the codebase.
Disable a handful of flake8's complaints when the style seems popular
enough among the team members to ignore.
Signed-off-by: Marti Bolivar <marti@foundries.io>
To properly test the project commands, it would be best to have a
fresh west bootstrapper package created and installed on PATH, so it
could be used to run commands exactly as they'd happen if we package
and ship the working tree.
To make that easier, add a dependency on tox and use it for testing:
https://tox.readthedocs.io/en/latest/
From now on, we'll test west by running 'tox' from the repository
root. This has several advantages over running pytest directly:
- "Just run tox": there are no longer any differences in test invocation
between POSIX OSes and Windows.
- tox creates an sdist package of the current tree using our setup.py
and installs it into a new virtual environment, then runs tests
there. This removes interference from other packages installed on
the host (like released bootstrappers that are also installed)
- we get to run multiple shell commands in order, should that ever be needed,
in our test procedures in a way that won't affect users
With that done, we can re-work the multirepo command testing to invoke
the bootstrapper in the virtual environment, adding various tests and
filling in longstanding testing gaps by adding increased checking of
the results (currently, much of the testing just checks whether
commands do or do not error out, which isn't enough).
These changes were made with a view towards the upcoming changes which
are planned before releasing west "into the wild": the test case code
should be mostly the same before and after the changes, so this serves
as a good baseline against regressions introduced by those upcoming
changes.
Signed-off-by: Marti Bolivar <marti@foundries.io>
[wip] debugging shippable results
Signed-off-by: Marti Bolivar <marti@foundries.io>
[wip] just test one py3
shutil.which west is picking up a 3.4 version in the 3.6 test, oddly
Signed-off-by: Marti Bolivar <marti@foundries.io>
This commit uses url-base instead of url to avoid confusion for users
as url in west section of manifest is full url, where url-base in manifest
section will be concatenated with project name to for clone url.
Signed-off-by: Torsten Rasmussen <torsten.rasmussen@nordicsemi.no>
In order to simplify and add consistency to the parsing of the manifest
file, represent the west section and the special project it represents
by using a subclass of the standard Project class.
Signed-off-by: Torsten Rasmussen <torsten.rasmussen@nordicsemi.no>
After switching from command-line parameters to a special section in the
manifest file, rework the tests so that they verify the new scheme of
things and add a new invalid manifest that can be tested in order to
verify that the schema qualification is working correctly.
Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
Create some dummy manifest, west, and project upstream repositories, and
a simple configuration file that points to them. Use this setup to test
--reset/update-manifest, --reset/update-west, and --reset-projects.
The --reset-* flags are used by 'west init' when reinitializing an
existing west installation. Also test that the correct --reset-* flags
are generated when calling the bootstrapper to reinitialize an
installation.
Some simple helper functions were added for creating repositories,
adding commits, and getting the subject of a repository's HEAD commit.
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
Simplify the projects tests to make them easier to understand and work
with:
- py.path (https://py.readthedocs.io/en/latest/path.html) functions
like mkdir() return the resulting path, so operations can be chained
- py.path paths have a write() method. Use it to write the manifest.
Get rid of the separate file.
- Switch to the following directory layout, which is simpler and makes
the manifest easier to locate:
<tmpdir>/repos/{kconfiglib,net-tools}
<tmpdir>/west/.west_marker
<tmpdir>/manifest.yml
<tmpdir> is used directly as the working directory.
- Add some more comments
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
Make commands that accept project names also accept project paths. For
example, 'west branch fix-foo .' can be run inside the zephyr/
directory, instead of running 'west branch fix-foo zephyr'.
Paths are mapped to projects by checking if the realpath (canonicalized
absolute) of any project is a prefix of the realpath of the supplied
path. That method will map subdirectories of projects to the project as
well.
Switch to using realpath() to calculate project.abspath as well, which
streamlines things and gets rid of gotchas related to symbolic links.
Project names and paths can be mixed in the same command. If both a
project name and a path matches, the name takes precedence (this can
always be worked around with './foo').
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
Now that we can list west and the manifest, it's worth restricting the
project namespace to prevent overlap. Note that it's still okay to
name a remote "west" or "manifest".
Add testing to make sure this doesn't regress.
Signed-off-by: Marti Bolivar <marti@foundries.io>
The mimxrt1052 does not have any internal flash, therefore a reset after
load blows away the code when loaded into sram. Reverse the order of the
pyocd commands such that the load follows the reset.
The command order now matches jlink.
Signed-off-by: Maureen Helm <maureen.helm@nxp.com>
On Windows shlex.split() will remove all backslash separators
unless specifically instructed.
Quote the path to the manifest before splitting so that
backslash path separators don't get removed on Windows.
Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
Tests were failing on Windows due to not using normalized paths.
Use normalized paths everywhere to avoid wrong path comparisons.
Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
I got tired of typing 'list-projects' all the time.
'list' might be a bit less obviously related to projects, but I don't
think it would be that bad.
Signed-off-by: Ulf Magnusson <ulfalizer@gmail.com>
This doesn't affect the default functionality of the runner, but it's
nicer to flash hex files if possible. For example, some tools find it
easier to post-process hex files than binary files for flashing on the
board.
Keep test cases up to date.
Signed-off-by: Marti Bolivar <marti@foundries.io>
Signed-off-by: Michael Scott <michael@foundries.io>
While implementing a manifest-based scan for the zephyr project, I
noticed that we aren't ensuring that projects always have distinct
paths (it's generally OK for them to have non-overlapping names; they
just can't be cloned to the same directories on the file system).
Add a check for that in west.manifest, along with a regression test.
Signed-off-by: Marti Bolivar <marti@foundries.io>
Currently, west.manifest mostly can just return a list of Project
types, each of which is a namedtuple for the project contents we care
about.
It would be better and more flexible to have a "real" data type for
the manifest, along with concrete representations for its contents
such as default values and remotes.
Reimplement the module to add just these things, adding Manifest,
Defaults, and Remote types, and re-working the Project namedtuple into
a class type. All of these (except Manifest) use __slots__ to restrict
their attributes.
Keep the current users and test cases up to date, extending the tests
a bit more as well. This moves a bit more code out of project.py into
west.manifest.
Signed-off-by: Marti Bolivar <marti@foundries.io>
Exercise both invalid manifest detection and correct parsing of valid
manifests. To make it easier to write tests for invalid manifests,
just glob for any invalid_*.yml files in the same directory as the
test file.
Signed-off-by: Marti Bolivar <marti@foundries.io>
We are getting user feedback about missing dependencies (like
"commands" and "log") that are actually imports of internal west
modules by those names.
We've been playing games with the python path to omit typing "west",
but this always caused inconsistency (as the bootstrapper version is
imported as west._bootstrap.version) and now it's causing outright
user confusion.
Let's just be consistent and import everything as west.xxx.
Fixes#57.
Signed-off-by: Marti Bolivar <marti@foundries.io>
The self-update mechanism checks the West and manifest repositories in
west/ for upstream changes and rebases them to the latest version if any
are found.
'west fetch' and 'west pull' run a self-update before fetching any
project data. It probably makes sense to always work with the latest
version of West and the manifest there. The self-update can be
suppressed with --no-update.
There's also a 'west update' command for manually updating West and/or
the manifest.
West automatically restarts itself after a self-update, before running
any commands or parsing the manifest. This makes it safe to add a new
feature to West and immediately start using it in the manifest.
The restarting is done with a plain exec(3). Getting this to work
required a few changes:
- West now uses absolute instead of relative imports, to make it
possible to execute src/west/main.py directly. This also makes it a
bit simpler to work on West. Relative imports seem to be discouraged
when googling around a bit.
__init__.py was removed from src/west. West is more of an application
than a package now (though it uses some packages internally).
- The internal 'cmd' package was renamed to 'commands', to avoid
clashing with a module in the Python standard library.
'runner' was renamed to 'runners' as well, for consistency.
The tests now have to be run from the West repository root with
'PYTHONPATH=src/west pytest'. The previous testing setup "just working"
relied on having run 'pip3 install -e .' first, which happened to add
src/west to the path due to the bootstrap script being there.
__init__.py was removed from the test directories as well. Packages
mostly make sense for tests if there are naming collisions, and naming
collisions will be obvious if packages aren't used.
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
This doesn't affect the behavior of the current west, but will be
needed to test an upcoming patch which creates and uses that file.
Signed-off-by: Marti Bolivar <marti@foundries.io>
Change the way the project test case fixtures are set up to create
local git repositories in the temporary directory, and a manifest that
refers to them.
This lets the test cases run without a network connection, which is
faster, and avoids false negatives that occur when the test
environment has a bad network connection.
Signed-off-by: Marti Bolivar <marti@foundries.io>
I was confused by the other design, because I intuitively expected West
metadata to go in .west/. To me the other design feels a little bit like
having an empty /home/foo/user/ directory that marks the /home/foo/
directory as containing the user's files.
Use west/ instead of .west/ to avoid "hiding" the implementation of the
building/flashing commands. I would've preferred to split them from the
repo-like functionality instead, but if they're going to be in the same
repository, then it's a decent comprise.
Remove the code that hides the west directory on Windows (via 'attrib')
as well, since it doesn't make much sense anymore.
This commit also fixes a small bug in west_topdir(): A west/ in the root
directory wasn't found.
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
Analogous to running 'git branch', except for all projects. Also lists
the project(s) each branch appears in.
Example output:
$ python3 -m west branch
FIX-FOO zephyr, Kconfiglib
manifest-rev zephyr, net-tools, Kconfiglib
master zephyr, net-tools
zephyr Kconfiglib
I'm planning to change 'git clone' into 'git init' + 'git remote add' +
'git fetch' later, which avoids creating the last two local branches.
Maybe manifest-rev should be hidden as well.
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
Passes its command as-is to the shell, within the repositories of each
of the specified projects (or all cloned projects by default).
I was thinking of making the shell quoting for long commands optional,
by joining all the words with a space or the like, but it might get
tricky with argparse. Could look into it more later.
Piggyback some cleanup: Make the absolute path of a project available in
project.abspath, to avoid having to do a bunch of join()s with
util.west_topdir().
Fix some copy-paste comment mess-ups in the tests too.
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
Add simple Google repo-like functionality, for dealing with multiple Git
repositories.
The command set is mirrored after Git. All commands except
'list-projects' accept an optional list of projects, and default to all
(cloned) projects.
- west list-projects
Lists projects and their repositories
- west fetch
Clone/fetch projects. Supports 'clone-depth', for making shallow
clones.
- west rebase
Rebase local branches to the revision specified in the manifest
- west pull
'west fetch' + 'west rebase' (similar to 'sync' in Google repo)
- west branch
Create a new branch in one or more repositories (for working on some
issue)
- west checkout
Check out a branch in each repository that has it. Supports a -b
flag for creating the branch first.
- west diff
Run 'git diff' in each repository
- west status
Run 'git status' in reach repository
There's no way to submit a multi-repository change for review yet.
Currently, a convenience branch 'manifest-rev' is created in each
project, which points to the revision specified for the project in the
manifest. 'manifest-rev' is updated by 'west fetch' and 'west pull'.
Local branches created with 'west branch' are set to track
'manifest-rev'. This makes e.g. a plain 'git status' or 'git pull' work
sensibly even when the manifest revision is an SHA.
We'll see if 'manifest-rev' is too magic later. It's explained in the
help texts of all the relevant commands at least.
It might be nicer to create 'manifest-rev' in e.g. refs/remotes instead
of refs/heads. Git doesn't seem to like trying to create a branch that
tracks a branch in refs/remotes unless it's a "proper" upstream branch
though.
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
Add an exhaustive test suite.
Check expected results for all combinations of runner constructor
parameters, both when instantiating the runner directly via its
constructor and when using command line arguments via its create()
method. Ensure consistent results.
Signed-off-by: Marti Bolivar <marti@foundries.io>
Add test coverage for flash, debug, and debugserver commands, using
runners created directly with their constructors, as well as via
command line arguments, with methods that would run syscalls mocked
out.
Signed-off-by: Marti Bolivar <marti@foundries.io>
Follow along with what the cool kids are doing and add py.test
integration.
This patch doesn't add any test cases, but sets up packages to
test the runner classes as additional work.
To run the test suite, use:
$ python3 setup.py test
Since there are no tests, these pass.
Signed-off-by: Marti Bolivar <marti@foundries.io>
Marti Bolivar