We need to use commit peeling to ensure that a revision which is a tag
is converted to a SHA. The current way Project.sha() works only
handles branches.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
The 'west status' output is no longer useful with many projects in the
mix. Apply a similar cleanup to 'west status' that 'west diff' got in
a53ec10cf2 ("west diff: only print
output for projects with nonempty diffs").
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This allows the user to always use unbuffered mode without having the use the
CLI option for every invocation of `west update`.
Signed-off-by: Michael Zimmermann <sigmaepsilon92@gmail.com>
these are copies of `test_update_projects` except that they pass job parameters
to `west update`.
There may be ways to test this feature in greater depth but right now this is
all I can think of.
Signed-off-by: Michael Zimmermann <sigmaepsilon92@gmail.com>
I missed this feature after switching from `repo` to `west`.
Depending on the number and size of projects, internet speed and IO speed this
can speed up syncing a lot. Smaller projects may not notice any difference.
Signed-off-by: Michael Zimmermann <sigmaepsilon92@gmail.com>
Recent versions of mypy have learned that the yaml module has type
stubs and the tool is now erroring out when it discovers we import
yaml since the stubs are not involved.
This is breaking CI on unrelated patches; fix it following the
instructions here:
https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
Based on: 8875340db4
Signed-off-by: Michael Zimmermann <michael.zimmermann@grandcentrix.net>
When a project revision in west.yml is a lightweight tag,
'git cat-file -t <revision>' prints 'commit' instead of 'tag'.
Hilarious as this may be, it confuses the _rev_type() helper used by
west update. The helper doesn't know what kind of revision it is,
since it's not a branch and doesn't look like a commit either. We then
fetch the tag unnecessarily since we don't know what we're dealing
with. That works, but we don't need to hit the network in this case.
Fix this by handling the lightweight tag case as well.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
The documentation says that the WEST_CONFIG_LOCAL environment variable
must override the default file location, <topdir>/.west/config, if
set. It doesn't, though; fix that.
This requires us to be a bit more careful in how we write the fixture
for the west.configuration test suite, and requires updating a test
case to match the documentation.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Add type annotations to existing APIs before extending this class.
This requires a slight tweak to the manifest property to avoid a
typing issue (https://github.com/python/mypy/issues/3004).
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
I ran into this testing 0.11.0a1. Our current code matches against the
last word in the 'git --version' output. Unfortunately, for 'git
version 2.24.3 (Apple Git-128)', that's incorrectly matching against
'Git-128)', which doesn't work.
Generalize the version detection a bit to handle this, and add
regression tests.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This will keep the URL of each submodule up to date if they change.
Allow users to get the old behavior via a new update.sync-submodules
config option, which defaults to True but can be set to False.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Silences the following, verbose and repetive warning quoted below when
using git >= 2.28
As this warning is 10 lines long and there are 40 projects in zephyr
right now, this cuts stderr for `west init` almost in half, from ~950
lines down to ~550 lines.
The non @static-ness of the git methods explained in PR #494 has to
"leak" to ensure_cloned(), at least for now. A single Update object
downloads all projects so this does not cause git --version to be
invoked many times.
hint: Using 'master' as the name for the initial branch. This default branch name
hint: is subject to change. To configure the initial branch name to use in all
hint: of your new repositories, which will suppress this warning, call:
hint:
hint: git config --global init.defaultBranch <name>
hint:
hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
hint: 'development'. The just-created branch can be renamed via this command:
hint:
hint: git branch -m <name>
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
These set default values of the --name-cache and --path-cache command
line options, respectively.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
We are setting up various bits and pieces of instance state before
getting to the meat of the work. This is getting to be enough lines
that a dedicated helper feels right.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Updating manifest-rev is a separate operation from fetching, but the
two are tracked and printed together in the --stats output.
Now that this code is in an instance method that has access to the
stats dict, it is easy to track manifest-rev and fetching separately,
so make it happen. Also resolve manifest-rev to a SHA before calling
_update_manifest_rev(), so the reflog entry says:
west update: moving to <SHA>
instead of something like:
west update: moving to FETCH_HEAD^{commit}
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This option tells west to fetch project.revision exactly, and nothing
else.
This skips fetching tags, and tries this even if the revision looks
like a SHA, which does not work depending on the git host.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This can be used to influence the way west fetches.
The motivating use case is to fetch with --depth=1. Since that's the
case, the new option can be used to override any 'clone-depth'
specified in the manifest for a project. Remove the extra 'with
--depth=foo' in the output when the project has a clone depth
accordingly, as this will be confusing if the user overrides it.
However, this is only part of the story to make this optimization
truly useful. We also need to allow the user to fetch something
that might be a SHA directly.
Without that, running 'west update -o --depth=1' on a project with
a SHA as the revision will sync the entire remote ref space, including
all tags, with --depth=1. Not quite what we're hoping for in terms of
limiting network bandwidth in that case.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Update.do_run() stashes the command line arguments in self.args
almost before doing anything else. There's no need to pass it around
or repeat that work.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This moves the definition closer to the point of use, allows us to
move the --stats bookkeeping into the method itself, and makes it more
obvious that 'update' is the only command that ought to be hitting the
network to fetch revisions.
For now, we don't need any instance data, but that will change in a
subsequent patch.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This allows the user to clone projects for the first time from
existing local repositories instead of fetching from the network.
For an example, consider this west.yml:
manifest:
defaults:
remote: myorg
remotes:
- name: myorg
url-base: https://github.com/myorg
projects:
- name: foo
path: modules/foo
revision: deadbeef
- name: bar
path: modules/bar
revision: abcd1234
Suppose the 'foo' and 'bar' repositories are already available in a
/var/my-name-cache directory, like this (.git locations shown for clarity):
/var/my-name-cache
├── bar
│ └── .git
└── foo
└── .git
You can then run:
west update --name-cache /var/my-name-cache
And the 'foo' and 'bar' projects will be cloned from '/var/my-name-cache/foo'
and '/var/my-name-cache/bar', respectively, before being updated.
NOTE:
It's /var/my-name-cache/foo, NOT /var/my-name-cache/modules/foo.
Project names are unique in a west manifest, so putting them in the
top level directory is safe. Doing it this way lets you use
--name-cache without duplicating cache repositories if project
paths might move around.
By contrast, if you had this /var/my-path-cache directory:
my-path-cache
└── modules
├── bar
│ └── .git
└── foo
└── .git
You could run:
west update --path-cache /var/my-path-cache
And get a similar result: the 'foo' and 'bar' projects will be cloned
from '/var/my-path-cache/modules/foo' and
'/var/my-path-cache/modules/bar', respectively.
Using either option, west update won't hit the network at all if the
'deadbeef' and 'abcd1234' SHAs are already available locally and
the (default) '--fetch=smart' strategy for west update is used.
If both options are given, both caches are checked; --name-cache is
checked first.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
It's been bugging me that we are making assumptions about the git
version without proper checks for a while.
Now we can start using WestCommand.git_version_info where appropriate
instead of making assumptions.
I still honestly have no idea what version of git is the minimum
required by west (we always make sure the version on the latest Ubuntu
LTS passes the test suite, providing some hint). That's unfortunate,
but we can start being more careful now to implement fallbacks if we
use 'new' features.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This allows us to reuse the same '_git' attribute already
added to WestCommand.git_version_info.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Commit c4a3bb8cb7 moved this helper from
manifest.py without adding a user for it. Remove it; the only thing we
are actually using is die_if_no_git().
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
On Windows, shlex.split('foo C:\\bar\\baz') results in:
['foo', 'C:barbaz']
This is not what we want; C:\\bar\\baz is a path and backslashes
should be preserved.
This has the result of making this test lead to different results
depending on whether the manifest URL is used by west as it runs or
not.
We could add posix=False to the shlex.split() call on Windows to avoid
this, but it's easier and more readable to just split the arguments
ourselves.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Instead of hard-coding a default revision to fetch from the remote,
get it from the remote URL itself if it's not provided by the user.
If that fails, we ask for the --manifest-rev option to be provided
explicitly.
This makes the code more robust by allowing each remote to dictate
policy instead of making assumptions. It is especially important as
many projects use 'main' instead of 'master' now.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This fix is present in v0.10-branch, but in the wrong commit due to a
botched rebase. Fix it.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
For schema version 0.10:
Allow group-filter values from imported manifests to affect the top
level manifest group filter. Simplify the results so that
Manifest.group_filter is just a list of disabled groups.
Manifests have the same precedence on the final group filter as they
do on project definition: manifests which appear earlier in the import
order have higher precedence on the group filter, just as they have
higher precedence when it comes to whether a project definition comes
from them or is ignored.
Delay loading the manifest.group-filter value from the local
configuration file until we actually need to know its value in
is_active().
Preserve 0.9 semantics if that is explicitly requested, but warn if
there are group filters anywhere.
Add test cases for the updated group-filter behavior, and regression
tests to ensure that 0.9 semantics are preserved when explicitly
requested.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This will be required by a later patch which uses the value '0.9'
explicitly to see if the user is requesting group-filter behavior that
we're going to change. It is also just a good way to tighten up the
error handling.
The intent here is to reject '0.9.99' as an invalid schema version, to
make sure nobody tries to do that. We're bumping the schema from 0.9
to 0.10 as part of the v0.10.0 development branch. Any unreleased
master branches during that time will have to request schema version
0.10 instead of 0.9.99 or something like that.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Move RawGroupType right before its only point of use.
Move GroupFilterType and GroupsType to the top level list of type
aliases, as prep work for using them in more places. Change the '#:'
comment style to '#' while we're here; the '#:' style is reserved for
comments which are also docstrings that end up in Sphinx, which
doesn't apply here.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Add more source code comments describing this internal class's
purpose. Use kwargs for fields whenever initializing an instance.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Change some things related to import map names:
- rename the _import_ctx 'filter_fn' attribute to 'imap_filter',
along with local variables using the same name
- rename _filter_ok() to _imap_filter_allows()
- rename _and_filters() to _compose_imap_filters()
- rename _new_ctx() to _compose_ctx_and_imap()
- adjust the debug logging when an import map filter rejects a project
This helps disambiguate filtering on projects that is due to import
map allowlists and blocklists versus project group filters.
These are all internal names: there are no API changes here.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Freezing, resolving, etc. a manifest file should result in output that
reflects the input's group-filter value too, but it doesn't. Fix that.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
It's annoying to have to specify projects lists in some cases
when all you're trying to do is exercise something else.
Though the number of situations where a manifest with no projects is
useful may be limited, they nonetheless exist, so make it possible.
This is a backwards incompatible change, so bump the schema version.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Martí Bolívar