This was requested by a user in a comment in issue #144.
The purpose is to make it easy to run commands on interesting subsets
of projects in a convenient way.
See help text for details.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Add a regression test for
https://github.com/zephyrproject-rtos/west/issues/545.
Mark it xfail for now since we don't have a fix. We'll remove that
once we have the fix in place.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Add test cases for this option, making sure not to use it in every
situation where we run west update with submodules to maintain
coverage for the case where it's not given.
Note that this is actually a work around for the test suite failing
using recent versions of git as well.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This is a work-around for:
https://github.blog/2022-10-18-git-security-vulnerabilities-announced/#cve-2022-39253
Our test cases are failing since they are using 'git submodule add' on
a URL which is a file. Recent versions of git are now rejecting this
by default due to the above CVE. We can override this default behavior
on a per-git-command basis by overriding the configuration option for
the duration of that process. There is no risk here since the
associated repositories are not maliciously crafted.
This work-around follows a suggestion here:
https://bugs.launchpad.net/ubuntu/+source/git/+bug/1993586/comments/3
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Test that west leaves the local repo alone when HEAD~0 is used.
HEAD~0 is introduced in the docs as the canonical way of doing this.
HEAD~0 is used instead of HEAD because HEAD causes west to instead fetch
the remote HEAD.
Signed-off-by: Øyvind Rønningstad <oyvind.ronningstad@nordicsemi.no>
Commit c757d6650f
("configuration: add Configuration class") was part of a general
re-work of how configuration file handling went.
As part of this change, an internal _whence() method was added, which
returns a list of configuration file paths on the file system for a
given ConfigFile enumerator. _whence() is currently erroring out with
a RuntimeError() when a local configuration file is requested, but not
found.
This breaks error handling in some cases.
For example, consider a flow like this:
# This command fails for some reason, leaving a .west
# directory but no .west/config
$ west init
# This command bombs out with RuntimeError: a topdir is
# found, so main.py tries to create a Manifest, which asks
# for the 'manifest.path' configuration option from the
# local file, which ... boom
$ west list
It would be better to raise MalformedConfig from here instead. This
lets higher layers handle this error better. In the above case,
we now get this instead:
$ west list
FATAL ERROR: can't load west manifest
local configuration file not found
This is clearly better. There are probably still some other error
handling edge cases that aren't being handled properly as a result of
this change, but I'd be curious to know how many of them this fixes.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Without a file argument, we should still use fall_back=True to search
for the workspace in the environment. This behavior was removed in the
recent rework of path handling, introducing a regression where the
manifest repository cannot be found when outside the workspace. Fix
it.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
(cherry picked from commit d275142374)
This is an API break. We are still allowed to do that until west 1.0.
The west.manifest.Manifest class's path handling has become
unmaintainable and we need to overhaul it. The from_data() and
from_file() constructors are too clever and try to allow too many use
cases, which results in the guts of the manifest construction code
often not being able to tell whether it's in a real workspace or not.
This makes it difficult to know when we can safely read configuration
files or not. That in turn makes it difficult to accurately
distinguish between the 'manifest.path' value in the local configuration
file and the 'self: path:' value in the manifest data, especially when
we can take keyword arguments that say "hey, pretend this is your
path".
Additionally, there are a few assumptions in the code that pretend
that the manifest file itself always lives in the top level directory
of the manifest repository. That was a valid assumption up to the
point the 'manifest.file' configuration variable was introduced, but
it's no longer valid anymore, since 'manifest.file' can point to a
subdirectory of the manifest repository. This leaves us with some
hacks to find the git repository we live in that shouldn't be
necessary and which we can now remove with this overhaul.
Rework the whole thing so that we can correctly handle the following
situations:
- when running in a workspace, you should now use from_file() or
a newly introduced from_topdir() to make your Manifest
- when running outside of any workspace, use from_data()
From now on, we forbid creating a manifest from data "as if" it were
in a real workspace. If all you have is data, the abspath attributes
will all be None, reflecting reality.
Accordingly, rework the Manifest.__init__() method to either take a
topdir or a bit of data, but not both. Additionally, now that we have
both manifest.path and manifest.file configuration options, it's not
usually the right thing to ask for a Manifest from a *file*: what you
really usually want is a Manifest from a *workspace*, and for that you
just need a topdir. From the topdir, you can create a
west.configuration.Configuration, and from there you can read
manifest.path and manifest.file to get the complete path to the top
level manifest within its repository. For that reason, we remove the
source_file keyword argument from __init__() in favor of topdir and a
new 'config' argument.
For backwards compatibility, it's still allowed to use from_file() to
load another manifest file than the one pointed to by
topdir/manifest.path/manifest.file. However, this handling code is now
just a bit of special case trickery in from_file(), and it also
introduces a restriction that the other file is still in a git
repository in the workspace. This moves potential sources for error or
confusion out of Manifest.__init__() and the functions it calls. (This
has proven to be a useful feature in practice; I am aware of west
extensions that use it to compare the parsed contents of two different
manifest files within the same workspace.)
To migrate away from the now-outdated from_file(), introduce a new
from_topdir() factory method and use it throughout the tree. This is
clearer and more accurate code, which always avoids the hacks left
behind for compatibility in from_file().
Finally, add various new instance attributes to the Manifest class
that will let us tell the YAML path from the actual path, and the path
to the manifest file from the path to the manifest repository in the
workspace. These in turn let us remove some hacky code from the 'west
list' implementation, and allow us to rework our internals so that
ManifestProject is just a legacy remnant at this point, further
helping along with #327.
Keep tests up to date, removing obsolete code as needed and updating
cases to reflect API breakage.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Commit 3f81eb4195
("manifest: project names must be unique") fixed an issue where
projects with duplicate names were previously permitted, but the test
case file that was meant to catch regressions was incorrectly left
empty. Fix it.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
It will be convenient in a future patch for west.manifest.validate()
to always return the validated dict which is loaded from YAML if all
we have is a str, so extend the behavior of the function by returning
the input dictionary it receives, or returning the dictionary it
parses.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Tweak some incidental details and add a couple of extra asserts.
Prep work for adding another test later on.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
The argparse module changed some implementation details in in a way
that's causing a false failure in testing. Handle it.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This isn't working properly because the ManifestProject's revision is
"HEAD", so it looks like an ordinary project and we try to get
manifest-rev from it. That is obviously incorrect. Fix it by detecting
the ManifestProject explicitly and doing the right thing there.
Fixes: #550
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This reverts commit 14b27b0940.
Parallel updates are a great feature, but they break color.ui and
attempts to fix that have caused unexpected exceptions to be thrown on
Windows, so we need to back the feature out for now until we can get
it done in a way that works with both color and Windows.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Make sure the userdata round-trips properly through 'west manifest
--freeze' by adding it to the project's dict representation.
Add a missing repr() inside the userdata related part of
Project.__repr__.
Fixes: #536
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Add a new optional 'userdata' key to projects.
The value will be parsed and stored in the userdata attribute of the
corresponding west.manifest.Project object.
Example manifest file:
manifest:
projects:
- name: foo
- name: bar
userdata: a-string
- name: baz
userdata:
key: value
defaults:
remote: r
remotes:
- name: r
url-base: base
Example Python usage:
manifest = west.manifest.Manifest.from_file()
foo, bar, baz = manifest.get_projects(['foo', 'bar', 'baz'])
print(foo.userdata) # None
print(bar.userdata) # prints 'a-string'
print(baz.userdata) # prints {'key': 'value'}
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Commit 6d42388f20
("tests: adjust create_repo() to keep working on newer git") got tox
working again in Arch on my laptop, and in CI for whatever GitHub
decides to use for ubuntu-latest, but it breaks tox on my Ubuntu 20.04
LTS testing environment that I use for release testing.
Fix it, again, and hopefully for the last time. See comments in the
source code for details on the fix.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
- names are not supposed to be paths
- it may be a mis-use of `repo-path`'s default value
- it would make using the name for log files harder
More info: https://github.com/zephyrproject-rtos/west/pull/515
Signed-off-by: Michael Zimmermann <sigmaepsilon92@gmail.com>
On the git v2.32.0 that's currently running on the Arch Linux install
I use for day to day development, I'm getting errors when I run tox.
This is because the test cases assume that newly created repositories
have 'master' branches, which isn't true in general anymore unless you
ask for it explicitly.
To keep things working for now, add a kwarg to the create_repo()
helper that preserves the old way of doing things by forcing a master
branch to exist.
Use 'git init --initial-branch' for this if that's available, but
otherwise just fall back on 'git checkout -b' for older versions of
Git to keep things consistent.
Since git itself is moving away from an assumption of a default branch
named 'master', we're going to need to do the same thing in west when
it comes to the default project revision. But that's a problem for
another day; for now let's just keep things duct taped together with
existing assumptions, since revisiting that will require a manifest
schema change.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This will be useful for creating new repositories in the test cases
with newer versions of git, which starting with v2.28 are moving away
from a default initial branch.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
these are copies of `test_update_projects` except that they pass job parameters
to `west update`.
There may be ways to test this feature in greater depth but right now this is
all I can think of.
Signed-off-by: Michael Zimmermann <sigmaepsilon92@gmail.com>
The documentation says that the WEST_CONFIG_LOCAL environment variable
must override the default file location, <topdir>/.west/config, if
set. It doesn't, though; fix that.
This requires us to be a bit more careful in how we write the fixture
for the west.configuration test suite, and requires updating a test
case to match the documentation.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
I ran into this testing 0.11.0a1. Our current code matches against the
last word in the 'git --version' output. Unfortunately, for 'git
version 2.24.3 (Apple Git-128)', that's incorrectly matching against
'Git-128)', which doesn't work.
Generalize the version detection a bit to handle this, and add
regression tests.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
These set default values of the --name-cache and --path-cache command
line options, respectively.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This allows the user to clone projects for the first time from
existing local repositories instead of fetching from the network.
For an example, consider this west.yml:
manifest:
defaults:
remote: myorg
remotes:
- name: myorg
url-base: https://github.com/myorg
projects:
- name: foo
path: modules/foo
revision: deadbeef
- name: bar
path: modules/bar
revision: abcd1234
Suppose the 'foo' and 'bar' repositories are already available in a
/var/my-name-cache directory, like this (.git locations shown for clarity):
/var/my-name-cache
├── bar
│ └── .git
└── foo
└── .git
You can then run:
west update --name-cache /var/my-name-cache
And the 'foo' and 'bar' projects will be cloned from '/var/my-name-cache/foo'
and '/var/my-name-cache/bar', respectively, before being updated.
NOTE:
It's /var/my-name-cache/foo, NOT /var/my-name-cache/modules/foo.
Project names are unique in a west manifest, so putting them in the
top level directory is safe. Doing it this way lets you use
--name-cache without duplicating cache repositories if project
paths might move around.
By contrast, if you had this /var/my-path-cache directory:
my-path-cache
└── modules
├── bar
│ └── .git
└── foo
└── .git
You could run:
west update --path-cache /var/my-path-cache
And get a similar result: the 'foo' and 'bar' projects will be cloned
from '/var/my-path-cache/modules/foo' and
'/var/my-path-cache/modules/bar', respectively.
Using either option, west update won't hit the network at all if the
'deadbeef' and 'abcd1234' SHAs are already available locally and
the (default) '--fetch=smart' strategy for west update is used.
If both options are given, both caches are checked; --name-cache is
checked first.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
On Windows, shlex.split('foo C:\\bar\\baz') results in:
['foo', 'C:barbaz']
This is not what we want; C:\\bar\\baz is a path and backslashes
should be preserved.
This has the result of making this test lead to different results
depending on whether the manifest URL is used by west as it runs or
not.
We could add posix=False to the shlex.split() call on Windows to avoid
this, but it's easier and more readable to just split the arguments
ourselves.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
For schema version 0.10:
Allow group-filter values from imported manifests to affect the top
level manifest group filter. Simplify the results so that
Manifest.group_filter is just a list of disabled groups.
Manifests have the same precedence on the final group filter as they
do on project definition: manifests which appear earlier in the import
order have higher precedence on the group filter, just as they have
higher precedence when it comes to whether a project definition comes
from them or is ignored.
Delay loading the manifest.group-filter value from the local
configuration file until we actually need to know its value in
is_active().
Preserve 0.9 semantics if that is explicitly requested, but warn if
there are group filters anywhere.
Add test cases for the updated group-filter behavior, and regression
tests to ensure that 0.9 semantics are preserved when explicitly
requested.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Freezing, resolving, etc. a manifest file should result in output that
reflects the input's group-filter value too, but it doesn't. Fix that.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
It's annoying to have to specify projects lists in some cases
when all you're trying to do is exercise something else.
Though the number of situations where a manifest with no projects is
useful may be limited, they nonetheless exist, so make it possible.
This is a backwards incompatible change, so bump the schema version.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Otherwise, setup functions that set git config variables like
user.name are potentially dangerous to run.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
There's just one test that's being skipped this way, and it's not that
slow compared to the rest.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Forcing users to define something unused is not just tedious, it's also
confusing.
Note the name attribute is mostly unused by git submodule too.
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
Some test cases are still referring to groups as things that can be
'allowed' and 'blocked'. We switched that language to 'enabled' and
'disabled' in code and documentation. Keep tests up to date.
Do the same in manifest-schema.yml.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
As a result of review of the project groups feature documentation,
change some of the names to be more clear, and be a bit more forgiving
in the error handling.
A project's 'groups' list remains the same. No changes there.
However, we make the following changes:
- 'manifest: groups:' is now 'manifest: group-filter:'
- we add a west.manifest.Manifest.group_filter value, exposing
'manifest: group-filter:' after validation and string conversion
- the config option 'manifest.groups' is now 'manifest.group-filter'
- invalid values in manifest.group-filter are downgraded to warnings
- the west update '--groups' option becomes '--group-filter', with
a slightly shorter name '--gf' if you can't handle that
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
During review of the documentation for the 0.9 release, the following
changes to manifest groups were requested:
- groups should be 'enabled' and 'disabled', not 'allowed' or
'blocked'
- enabling a group should be done by prefixing its name with '+'
in "manifest: groups:" or manifest.groups
Projects are still 'active' or 'inactive'.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
If the manifest repository path is placed in a nested subdirectory of
the workspace, shutil.move() does not guarantee success. Strangely,
this works on Linux, even though it probably shouldn't, but it fails
on Windows. Fix that and add a regression test.
Fixes: #463
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Martí Bolívar