To properly test the project commands, it would be best to have a
fresh west bootstrapper package created and installed on PATH, so it
could be used to run commands exactly as they'd happen if we package
and ship the working tree.
To make that easier, add a dependency on tox and use it for testing:
https://tox.readthedocs.io/en/latest/
From now on, we'll test west by running 'tox' from the repository
root. This has several advantages over running pytest directly:
- "Just run tox": there are no longer any differences in test invocation
between POSIX OSes and Windows.
- tox creates an sdist package of the current tree using our setup.py
and installs it into a new virtual environment, then runs tests
there. This removes interference from other packages installed on
the host (like released bootstrappers that are also installed)
- we get to run multiple shell commands in order, should that ever be needed,
in our test procedures in a way that won't affect users
With that done, we can re-work the multirepo command testing to invoke
the bootstrapper in the virtual environment, adding various tests and
filling in longstanding testing gaps by adding increased checking of
the results (currently, much of the testing just checks whether
commands do or do not error out, which isn't enough).
These changes were made with a view towards the upcoming changes which
are planned before releasing west "into the wild": the test case code
should be mostly the same before and after the changes, so this serves
as a good baseline against regressions introduced by those upcoming
changes.
Signed-off-by: Marti Bolivar <marti@foundries.io>
[wip] debugging shippable results
Signed-off-by: Marti Bolivar <marti@foundries.io>
[wip] just test one py3
shutil.which west is picking up a 3.4 version in the 3.6 test, oddly
Signed-off-by: Marti Bolivar <marti@foundries.io>
This prevents random wests in developer PYTHONPATHs from coming
first. It's being done for convenience during development and testing.
Signed-off-by: Marti Bolivar <marti@foundries.io>
Git clone can take a raw file system path instead of a URL when
cloning a repository that is already available locally somewhere else.
When this happens, git tries to use hard links to create the files in
the destination, which is more efficient, so the limitation to URLs is
not good. Remove it.
Signed-off-by: Marti Bolivar <marti@foundries.io>
The _expand_shorthands() helper function allows constructing strings
based on attributes of a project. However, it uses a custom format
string DSL-like syntax instead of picking one of the two which already
exist in Python.
Convert this to use the standard .format() string formatter.
This is mostly a mechanical change. The only thing worth calling out
is that when using .format() to construct something that will get
passed to _expand_shorthands(), we need to double the brackets for the
shorthand keys (so "{{" / "}}" instead of "{" / "}") so that the first
.format interprets them as ordinary brackets, which the call to
.format() in _expand_shorthands() then sees as an access to a format
value by name.
Signed-off-by: Marti Bolivar <marti@foundries.io>
Miscellaneous fixes and improvements to the manifest parsing code.
- remove unused and undocumented 'url' parameter to Project
constructor: this only matters for SpecialProject instances
- add and fix pydoc parameter documentation for a few classes
- following manifest schema changes, rename the Remote 'url' parameter
to 'url_base'
- NotImplemented is not an exception; it should be returned, not
raised
- pass project paths through os.path.normpath() when initializing
Project instances
Signed-off-by: Marti Bolivar <marti@foundries.io>
We need to do another copy of west into Zephyr to get some runner
fixes and features, but some assumptions have crept back into the code
that we're in a multirepo install.
Fix these so we can use flash and debug commands in a monorepo Zephyr
installation.
Signed-off-by: Marti Bolivar <marti@foundries.io>
On Microsoft Windows, the default name of the JLink executable is
"JLink.exe", while on *NIX it is instead "JLinkExe". Detect and
correctly set the default name based on the platform.
Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
Fixes: #139
Fixes the issue of fetching SHAs when specified in manifest file.
It is checked if a revision is a SHA, in which case the fetch is
performed on the url and afterwards the manifest-rev reference is
updated to the SHA.
Signed-off-by: Torsten Rasmussen <torsten.rasmussen@nordicsemi.no>
Fixing the behavior of 'west debug' requires users to update their
bootstrap installation. Bump the minor release number to make that
possible.
Signed-off-by: Marti Bolivar <marti@foundries.io>
Commit f8bd521 ("Add self-update for West and the manifest") switched
from importing the west module and invoking main directly from the
same process as the bootstrapper to invoking it as a subprocess.
This breaks 'west debug'.
Runners which run GDB directly to debug a target need to be able to
ignore SIGINT while GDB is running, as Ctrl-C halts the target and
returns control to the (gdb) prompt instead of terminating the program
as usual.
Because of that, the runner code must be running in the same process
as the bootstrapper, or the parent (bootstrap) process, which doesn't
ignore the signal, means the control C at the shell gets delivered to
a process which uses the default handler to exit the program.
Fix it by going back to the mechanism the bootstrapper originally
used to delegate a command to the installed west.
Fixes: #135
Signed-off-by: Marti Bolivar <marti@foundries.io>
The argument array to main is in the argv argument, *not* sys.argv.
Fix this.
(Using sys.argv directly prevents the main function from getting its
arguments from an arbitrary value specified by the caller.)
Signed-off-by: Marti Bolivar <marti@foundries.io>
It's not clear from the help how the source and build directories are
determined. Fix that and sentence-case the other help text while we're
here.
Signed-off-by: Marti Bolivar <marti@foundries.io>
Fixes: #119
This commit changes the way show-ref is executed in order to determine
whether HEAD is pointing to a valid reference or if the repository is
in initialized stated.
Signed-off-by: Torsten Rasmussen <torsten.rasmussen@nordicsemi.no>
This commit uses url-base instead of url to avoid confusion for users
as url in west section of manifest is full url, where url-base in manifest
section will be concatenated with project name to for clone url.
Signed-off-by: Torsten Rasmussen <torsten.rasmussen@nordicsemi.no>
In order to simplify and add consistency to the parsing of the manifest
file, represent the west section and the special project it represents
by using a subclass of the standard Project class.
Signed-off-by: Torsten Rasmussen <torsten.rasmussen@nordicsemi.no>
After switching from command-line parameters to a special section in the
manifest file, rework the tests so that they verify the new scheme of
things and add a new invalid manifest that can be tested in order to
verify that the schema qualification is working correctly.
Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
Since the metadata associated with west itself (url and revision) is no
longer provided via command-line parameters but rather using a special
section in the manifest file, convert project.py to make use of the new
capabilities in the manifest parsing code and drop the metadata from the
configuration file, where it no longer belongs.
Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
Rework the manifest parsing in order to support both the manifest and
west sections that can now be present in the yaml file.
This enables the manifest to be parsed fully and include both the
project information from the manifest section and also the special
metadata associated with west itself.
Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
In order to support encoding the required west URL and revision when
tagging a manifest for a release, move the metadata regarding west
itself (url and revision) from the command-line to the manifest.
This involves:
- Adding a new "west" section to the manifest that includes the metadata
- Removing the "-w" and "--wr" command-line options
- Parsing the "west" section of the manifest directly in the
bootstrapper
Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
Older versions of git lack support for --rebase=false
Thus splitting pull into a fetch and merge is safer, as well as it
provides better help on which steps are failing.
Signed-off-by: Torsten Rasmussen <torsten.rasmussen@nordicsemi.no>
The error output when the bootstrapper is called without arguments or
with -h is lacking. Try to improve it.
Tweak the output when called without arguments when an installation is
found as well, in the same spirit.
Finally, change the print_usage() to a print_help() when no command is
given, to get additional context.
Fixes: #108
Signed-off-by: Marti Bolivar <marti@foundries.io>
Create some dummy manifest, west, and project upstream repositories, and
a simple configuration file that points to them. Use this setup to test
--reset/update-manifest, --reset/update-west, and --reset-projects.
The --reset-* flags are used by 'west init' when reinitializing an
existing west installation. Also test that the correct --reset-* flags
are generated when calling the bootstrapper to reinitialize an
installation.
Some simple helper functions were added for creating repositories,
adding commits, and getting the subject of a repository's HEAD commit.
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
Simplify the projects tests to make them easier to understand and work
with:
- py.path (https://py.readthedocs.io/en/latest/path.html) functions
like mkdir() return the resulting path, so operations can be chained
- py.path paths have a write() method. Use it to write the manifest.
Get rid of the separate file.
- Switch to the following directory layout, which is simpler and makes
the manifest easier to locate:
<tmpdir>/repos/{kconfiglib,net-tools}
<tmpdir>/west/.west_marker
<tmpdir>/manifest.yml
<tmpdir> is used directly as the working directory.
- Add some more comments
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
Support rerunning 'west init' in an already initialized West
installation to update configuration values in west/config.
Only arguments that are explicitly passed are changed. For example,
'west init --mr foo' will update the manifest revision to 'foo', leaving
all other settings unchanged. Passing no arguments when reinitializing
is a no-op, and prints an error.
The initial west/config creation is treated as an update that happens to
set all configuration values, to reuse some code.
Updating the manifest URL or revision via 'west init' automatically
resets the manifest to the new revision (after fetching it). This is
implemented with a new 'west update --reset-manifest' flag. Updating the
manifest also resets all (cloned) projects, via a new
'west update --reset-projects' flag.
Updating the west URL/revision resets the west repository, via a new
'west update --reset-west' flag.
All resetting is done with 'git reset --keep', which is a bit safer than
'git reset --hard', as it never discards uncommitted changes.
The reset behavior can be suppressed by passing --no-reset to
'west init'.
This commit also adds more comprehensive documentation for the behavior
of 'west init'.
Piggyback some small cleanup:
- Rename WEST/MANIFEST_DEFAULT to WEST/MANIFEST_URL_DEFAULT. There's
already WEST/MANIFEST_REV_DEFAULT.
- Rename find_west_topdir() to west_topdir(). This is what the version
in the main West code is called (would be nice to reuse things
directly from there in the future).
Also make west_topdir() implicitly default to the working directory.
Fixes: #77
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
We intend that 'west -v flash' provides sufficient information for the
motivaed user to stop using west if they so choose. The use of a
temporary file in the jlink runner whose contents are deleted as west
exits prevents that. Print its contents so we can let people get back
to work without west.
Fixes: #113
Signed-off-by: Marti Bolivar <marti@foundries.io>
Make commands that accept project names also accept project paths. For
example, 'west branch fix-foo .' can be run inside the zephyr/
directory, instead of running 'west branch fix-foo zephyr'.
Paths are mapped to projects by checking if the realpath (canonicalized
absolute) of any project is a prefix of the realpath of the supplied
path. That method will map subdirectories of projects to the project as
well.
Switch to using realpath() to calculate project.abspath as well, which
streamlines things and gets rid of gotchas related to symbolic links.
Project names and paths can be mixed in the same command. If both a
project name and a path matches, the name takes precedence (this can
always be worked around with './foo').
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
Rename variables to make west more agnostic, and therefore potentially
more flexible, to what it is flashing.
This is a small step towards solving #7868 and other similair issues
in Zephyr. Essentially, sometimes, for instance when a bootloader is
used, or a dual-core platform is used, we need to flash multiple
images/application's, not just the 'kernel'.
To prepare 'west' for this future feature we make it agnostic to what
kind of hex/bin/elf file it is flashing by taking the 'kernel' name
out of the variables. This allows the hex file to contain not just the
kernel, but also a bootloader, or a second image.
Signed-off-by: Sebastian Bøe <sebastian.boe@nordicsemi.no>
Now that we can list west and the manifest, it's worth restricting the
project namespace to prevent overlap. Note that it's still okay to
name a remote "west" or "manifest".
Add testing to make sure this doesn't regress.
Signed-off-by: Marti Bolivar <marti@foundries.io>
Add a -f / --format option that allows users to list projects exactly how
they want them.
Also add a -a / --all flag that does not ignore the west or manifest
repositories in the listing. This is by analogy with the ls command;
we are treating west and the manifest as if they were hidden.
Signed-off-by: Marti Bolivar <marti@foundries.io>
Support a list of projects in the arguments. The listed projects will
be limited to those named, in the order they are given on the command
line. This is the same way that "fetch", "forall", and other commands
operate on all projects by default, or just specified projects.
Remove the special-case text printing the manifest path. We'll handle
that by treating west and the manifest as projects in a later patch.
Signed-off-by: Marti Bolivar <marti@foundries.io>
This creates extra slashes when users naively concatenate two paths
with the system path separator. It also looks weird on Windows.
Signed-off-by: Marti Bolivar <marti@foundries.io>
This reverts commit bc7b1c595ed4090be6d80859a01a0bb372ae030c in
the zephyr main tree to fix the runner script and also does the
below modifications.
1) explicitly terminate processes.
2) implicitly call debugserver when debug is called.
Signed-off-by: Rajavardhan Gundi <rajavardhan.gundi@intel.com>
The mimxrt1052 does not have any internal flash, therefore a reset after
load blows away the code when loaded into sram. Reverse the order of the
pyocd commands such that the load follows the reset.
The command order now matches jlink.
Signed-off-by: Maureen Helm <maureen.helm@nxp.com>
'git fetch/pull' by default doesn't fetch tags from remotes specified as
URLs, which is what we use. Pass --tags to get them.
This will only fetch tags pointing to commits reachable from the
manifest revision being fetched.
Fixes: #92
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
On Windows shlex.split() will remove all backslash separators
unless specifically instructed.
Quote the path to the manifest before splitting so that
backslash path separators don't get removed on Windows.
Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
Marti Bolivar