`enumerate` on `imap_unordered` does (obviously) not return the index of the
element that finished processing.
That resulted in printing the wrong log file which might not even be finished
and thus swallowing future logs of that job.
Instead, let each job return it's index.
Fixes: #528
Signed-off-by: Michael Zimmermann <sigmaepsilon92@gmail.com>
Commit 6d42388f20
("tests: adjust create_repo() to keep working on newer git") got tox
working again in Arch on my laptop, and in CI for whatever GitHub
decides to use for ubuntu-latest, but it breaks tox on my Ubuntu 20.04
LTS testing environment that I use for release testing.
Fix it, again, and hopefully for the last time. See comments in the
source code for details on the fix.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Updates for clarity and to match current practice.
Reflect some practical changes that I've been doing in the past couple
of release that cut down on review latency once it's been agreed that
it's release time. In particular, we are cutting out the review cycles
for forking the release branch, changing the versions, and tagging the
main branch after it's moved from the release branch.
Rework the section in to step 1-2-3 style while we're here, to make it
easier to follow along with.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Fixes: #523
The Manifest class has these instance attributes:
- _projects_by_name: maps project names -> Projects, has 'manifest' as
a key and the ManifestProject as a value (this is one reason why
'manifest' is a reserved project name)
- _projects_by_rpath: maps resolve()d Paths -> Projects, has the
manifest repository's path as a key, pointing to the ManifestProject
So get_projects() ought to just rely on these dicts to do its job.
However, it's currently got extra special-case code it doesn't need,
and the code is buggy. Specifically this part:
mpath: Optional[Path] = Path(mp.path).resolve()
This is trying to get the ManifestProject's resolved path, but it only
works if you happen to be in the workspace topdir, since mp.path is
relative.
We got "lucky" and can still find the manifest project by its relative
path because we fall back on _projects_by_rpath.
But this is still a correctness problem. When the current
directory *isn't* the topdir, but has a subdirectory or file inside it
whose name happens to match mp.path, we could return the
ManifestProject as a result, incorrectly.
Just remove all the unnecessary extra code and use _projects_by_name /
_projects_by_rpath to do the work.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
- names are not supposed to be paths
- it may be a mis-use of `repo-path`'s default value
- it would make using the name for log files harder
More info: https://github.com/zephyrproject-rtos/west/pull/515
Signed-off-by: Michael Zimmermann <sigmaepsilon92@gmail.com>
West init without --mr is leaving the manifest repository in a
detached HEAD state.
Fixes: #522
The convoluted logic we use to "clone" the manifest repository using
init and fetch is only in place to allow using a magic GitHub pull
request ref as --manifest-rev. This small benefit is no longer worth
the downsides.
To fix it, just use 'git clone' instead. This also lets us
drop a bunch of extra logic trying to figure out what the remote wants
us to use by default.
It also avoids getting the now-infamous "Using 'master' as the name
for the initial branch." message on the first west init if the user
doesn't have init.defaultBranch set.
Preserve 'west init --manifest-rev foo' using 'git clone --branch
foo'. This works fine with branches or tags, but you'll no longer be
able to initialize a workspace from a 'pull/1234/head' GitHub pull
request reference or a SHA. Affected users are going to need to do
'west init', then fetch the revision they need before running 'west
update'.
Unlike other branch related options (like 'git init --initial
branch'), we've had 'git clone --branch' since somewhere in the git
v1.x days, so it's safe to use unconditionally.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Now that git isn't necessarily creating that branch,
we should update our comments for correctness.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
On the git v2.32.0 that's currently running on the Arch Linux install
I use for day to day development, I'm getting errors when I run tox.
This is because the test cases assume that newly created repositories
have 'master' branches, which isn't true in general anymore unless you
ask for it explicitly.
To keep things working for now, add a kwarg to the create_repo()
helper that preserves the old way of doing things by forcing a master
branch to exist.
Use 'git init --initial-branch' for this if that's available, but
otherwise just fall back on 'git checkout -b' for older versions of
Git to keep things consistent.
Since git itself is moving away from an assumption of a default branch
named 'master', we're going to need to do the same thing in west when
it comes to the default project revision. But that's a problem for
another day; for now let's just keep things duct taped together with
existing assumptions, since revisiting that will require a manifest
schema change.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This will be useful for creating new repositories in the test cases
with newer versions of git, which starting with v2.28 are moving away
from a default initial branch.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
We need to use commit peeling to ensure that a revision which is a tag
is converted to a SHA. The current way Project.sha() works only
handles branches.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
The 'west status' output is no longer useful with many projects in the
mix. Apply a similar cleanup to 'west status' that 'west diff' got in
a53ec10cf2 ("west diff: only print
output for projects with nonempty diffs").
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This allows the user to always use unbuffered mode without having the use the
CLI option for every invocation of `west update`.
Signed-off-by: Michael Zimmermann <sigmaepsilon92@gmail.com>
these are copies of `test_update_projects` except that they pass job parameters
to `west update`.
There may be ways to test this feature in greater depth but right now this is
all I can think of.
Signed-off-by: Michael Zimmermann <sigmaepsilon92@gmail.com>
I missed this feature after switching from `repo` to `west`.
Depending on the number and size of projects, internet speed and IO speed this
can speed up syncing a lot. Smaller projects may not notice any difference.
Signed-off-by: Michael Zimmermann <sigmaepsilon92@gmail.com>
Recent versions of mypy have learned that the yaml module has type
stubs and the tool is now erroring out when it discovers we import
yaml since the stubs are not involved.
This is breaking CI on unrelated patches; fix it following the
instructions here:
https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
Based on: 8875340db4
Signed-off-by: Michael Zimmermann <michael.zimmermann@grandcentrix.net>
When a project revision in west.yml is a lightweight tag,
'git cat-file -t <revision>' prints 'commit' instead of 'tag'.
Hilarious as this may be, it confuses the _rev_type() helper used by
west update. The helper doesn't know what kind of revision it is,
since it's not a branch and doesn't look like a commit either. We then
fetch the tag unnecessarily since we don't know what we're dealing
with. That works, but we don't need to hit the network in this case.
Fix this by handling the lightweight tag case as well.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
The documentation says that the WEST_CONFIG_LOCAL environment variable
must override the default file location, <topdir>/.west/config, if
set. It doesn't, though; fix that.
This requires us to be a bit more careful in how we write the fixture
for the west.configuration test suite, and requires updating a test
case to match the documentation.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Add type annotations to existing APIs before extending this class.
This requires a slight tweak to the manifest property to avoid a
typing issue (https://github.com/python/mypy/issues/3004).
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
I ran into this testing 0.11.0a1. Our current code matches against the
last word in the 'git --version' output. Unfortunately, for 'git
version 2.24.3 (Apple Git-128)', that's incorrectly matching against
'Git-128)', which doesn't work.
Generalize the version detection a bit to handle this, and add
regression tests.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This will keep the URL of each submodule up to date if they change.
Allow users to get the old behavior via a new update.sync-submodules
config option, which defaults to True but can be set to False.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Silences the following, verbose and repetive warning quoted below when
using git >= 2.28
As this warning is 10 lines long and there are 40 projects in zephyr
right now, this cuts stderr for `west init` almost in half, from ~950
lines down to ~550 lines.
The non @static-ness of the git methods explained in PR #494 has to
"leak" to ensure_cloned(), at least for now. A single Update object
downloads all projects so this does not cause git --version to be
invoked many times.
hint: Using 'master' as the name for the initial branch. This default branch name
hint: is subject to change. To configure the initial branch name to use in all
hint: of your new repositories, which will suppress this warning, call:
hint:
hint: git config --global init.defaultBranch <name>
hint:
hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
hint: 'development'. The just-created branch can be renamed via this command:
hint:
hint: git branch -m <name>
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
These set default values of the --name-cache and --path-cache command
line options, respectively.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
We are setting up various bits and pieces of instance state before
getting to the meat of the work. This is getting to be enough lines
that a dedicated helper feels right.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Updating manifest-rev is a separate operation from fetching, but the
two are tracked and printed together in the --stats output.
Now that this code is in an instance method that has access to the
stats dict, it is easy to track manifest-rev and fetching separately,
so make it happen. Also resolve manifest-rev to a SHA before calling
_update_manifest_rev(), so the reflog entry says:
west update: moving to <SHA>
instead of something like:
west update: moving to FETCH_HEAD^{commit}
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This option tells west to fetch project.revision exactly, and nothing
else.
This skips fetching tags, and tries this even if the revision looks
like a SHA, which does not work depending on the git host.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This can be used to influence the way west fetches.
The motivating use case is to fetch with --depth=1. Since that's the
case, the new option can be used to override any 'clone-depth'
specified in the manifest for a project. Remove the extra 'with
--depth=foo' in the output when the project has a clone depth
accordingly, as this will be confusing if the user overrides it.
However, this is only part of the story to make this optimization
truly useful. We also need to allow the user to fetch something
that might be a SHA directly.
Without that, running 'west update -o --depth=1' on a project with
a SHA as the revision will sync the entire remote ref space, including
all tags, with --depth=1. Not quite what we're hoping for in terms of
limiting network bandwidth in that case.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Update.do_run() stashes the command line arguments in self.args
almost before doing anything else. There's no need to pass it around
or repeat that work.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This moves the definition closer to the point of use, allows us to
move the --stats bookkeeping into the method itself, and makes it more
obvious that 'update' is the only command that ought to be hitting the
network to fetch revisions.
For now, we don't need any instance data, but that will change in a
subsequent patch.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This allows the user to clone projects for the first time from
existing local repositories instead of fetching from the network.
For an example, consider this west.yml:
manifest:
defaults:
remote: myorg
remotes:
- name: myorg
url-base: https://github.com/myorg
projects:
- name: foo
path: modules/foo
revision: deadbeef
- name: bar
path: modules/bar
revision: abcd1234
Suppose the 'foo' and 'bar' repositories are already available in a
/var/my-name-cache directory, like this (.git locations shown for clarity):
/var/my-name-cache
├── bar
│ └── .git
└── foo
└── .git
You can then run:
west update --name-cache /var/my-name-cache
And the 'foo' and 'bar' projects will be cloned from '/var/my-name-cache/foo'
and '/var/my-name-cache/bar', respectively, before being updated.
NOTE:
It's /var/my-name-cache/foo, NOT /var/my-name-cache/modules/foo.
Project names are unique in a west manifest, so putting them in the
top level directory is safe. Doing it this way lets you use
--name-cache without duplicating cache repositories if project
paths might move around.
By contrast, if you had this /var/my-path-cache directory:
my-path-cache
└── modules
├── bar
│ └── .git
└── foo
└── .git
You could run:
west update --path-cache /var/my-path-cache
And get a similar result: the 'foo' and 'bar' projects will be cloned
from '/var/my-path-cache/modules/foo' and
'/var/my-path-cache/modules/bar', respectively.
Using either option, west update won't hit the network at all if the
'deadbeef' and 'abcd1234' SHAs are already available locally and
the (default) '--fetch=smart' strategy for west update is used.
If both options are given, both caches are checked; --name-cache is
checked first.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
It's been bugging me that we are making assumptions about the git
version without proper checks for a while.
Now we can start using WestCommand.git_version_info where appropriate
instead of making assumptions.
I still honestly have no idea what version of git is the minimum
required by west (we always make sure the version on the latest Ubuntu
LTS passes the test suite, providing some hint). That's unfortunate,
but we can start being more careful now to implement fallbacks if we
use 'new' features.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This allows us to reuse the same '_git' attribute already
added to WestCommand.git_version_info.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Commit c4a3bb8cb7 moved this helper from
manifest.py without adding a user for it. Remove it; the only thing we
are actually using is die_if_no_git().
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Michael Zimmermann