Git-gpush-scripts

The git-gpush Script Suite

This page contains some background information for the git-gpush, git-gpick, git-gpull, and git-ggc scripts found in the qt/qtrepotools repository.

Note: The features described herein are still under review. The first Change in the series is https://codereview.qt-project.org/c/qt/qtrepotools/+/152266

Overview

The general idea of these scripts is to make working with Gerrit more transparent:

• You can pull as often as you want, including right before pushing, as you would usually do in a git pull --rebase based workflow. Without git-gpush, this would unnecessarily rebase previously pushed Changes, which would make Gerrit's inter-diff feature slower and much noisier.
• You can keep many unrelated Changes locally without creating spurious dependencies between them on Gerrit. While you can achieve the same by having a separate local branch for every series, having all Changes in a single branch is much more convenient.
• You can push updates to specific Changes without invalidating reviews of unrelated Changes.

You may have noticed that these major features are geared towards making not your life easier, but that of your reviewers. Until roles are switched, that is.

The way how these scripts achieve this is by letting you manage "virtual branches" (called "series") within your local working branch. When a series is pushed, its commits are rebased (usually to a commit from the upstream branch) to isolate it from other local Changes. The working tree is not affected by this in any way, so no unnecessary rebuilds result from this.

More Features

git gpush

• Updates to the Changes pushed from other clones of the target repository (or made directly on Gerrit) are detected, so you won't accidentally overwrite them.
• The target branch of each series is remembered, server-side moves are tracked, and attempts to create duplicates by pushing for the wrong branch are rejected by default.
• The scripts remember which Changes belong to a series, so once you pushed it, the complete series can be identified by specifying a single Change (or none at all, if it's HEAD). This saves you from needing to keep track of how many Changes are in the series. You don't even need to use the currently last Change in the series. Also, commits can be specified by Change-Id. Taken together, these features make it possible to use the same command line to repeatedly re-push the same series regardless of any local history rewriting.
• Or you can just push out all local series in one go.
• An overview of the pushed Changes and their status relative to Gerrit is displayed.
• Pushes which would be rejected by Gerrit are detected upfront.
• Unsquashed fixup commits and merges of non-upstream commits are rejected.
• Changes duplicated between local branches are detected, and an authoritative source branch for actual pushes can be set.
• Reviewers/CCs can be conveniently added by their IRC nickname.

git gpick

This script allows you to download pending Changes into your local branch, for local testing, reviewing and amending.

• Notable differences from Gerrit's built-in copy&paste-able download command lines:
  • Unlike 'cherry-pick', it can download an entire series in one go, so you just need to copy the Change-Id of the last Change and execute a single short command.
  • Unlike 'checkout', your local Changes are not "hidden" on another branch, so you don't need to "switch contexts" (rebuild lots of stuff, deal with bugs you already fixed, etc.).
  • Unlike 'pull', no merge commit is introduced, so local history rewriting remains unproblematic.
• The downloaded Changes are a separate series, so there is no danger that you accidentally re-push them together with other Changes.
• Changes which already exist locally are replaced, so it is possible to pull in updates which have been pushed from other clones (or done directly on Gerrit).
  • This checks for conflicts, so you won't lose concurrent local modifications.
  • Merging of non-overlapping concurrent modifications is possible.
  • It's also possible to apply only certain parts of the updated Changes, e.g., the commit message.
  • There also exists the --check mode which only shows the updates on Gerrit without replacing any local commits.
    • This being a simple 3-way diff, it also displays local modifications with more granularity than git-gpush does.

Getting Started

As usual, you should start with some solid RTFM: ;)

$ git-gpush -h
$ git-gpick -h
$ git-gpull -h
$ git-ggc -h

Get all your pending changes onto the current branch, in case you kept them on separate branches:

$ git cherry-pick <…>

Alternatively, you could merge all your local branches and then rebase.

If you have pending Changes for the given branch on Gerrit, you need to synchronize the state (re-bootstrap):

$ git gpick --check --all

To get an idea about what gpush thinks you have in your branch:

$ git gpush --list

Group remaining loose Changes into series:

$ git gpush --group :3
$ git gpush --group ~3:5
<…>

Push out any modified Changes:

$ git gpush --all

If Something Goes Wrong ...

• if you get something that seems incomprehensible, slow down, think about it, don't use --force-* options right away
• if you get a crash or other behavior that cannot be reasonably explained, make a bug report:
  • make a JIRA task against QTQAINFRA/Git, and make sure it is assigned to me (Ossi), or contact me directly via freenode (ossi) or matrix/facebook/skype (ossilator)
  • always save the backtrace if you get one, obviously
  • add the recent relevant shell history (that is, git commands that modify something)
  • add a snapshot of the git/gpush state, created with:
    git bundle create gpush-state.bundle $(git for-each-ref --format='%(refname)' refs/gpush/) @{u}..
• if you manage to get the scripts so confused that none of the documented overrides help, you can throw away the complete state with:
  git update-ref -d refs/gpush/state
  note that it is very unlikely that you actually need to do that, and before you do, save a snapshot and make a bug report

Examples

To create a series consisting of two Changes, and push it, adding two reviewers:

<hack hack hack>
$ git commit -a
<hack hack hack>
$ git commit -a
$ git gpush :2 +someguy +anotherguy@example.com

Amend and re-push the series:

$ git rebase -i @{u}
<…>
$ git gpush

Add another Change to the series:

<hack hack hack>
$ git commit -a
$ git gpush -e

Re-push the series when it's not at the top any more:

$ git gpush I123415affe

Download somebody else's series:

$ git gpick +Ie3f7567639cc05e03257cd5bac133714e5c4f068

No need to worry about accidentally re-pushing it any more!

Update your local copy after somebody has been messing with your Change (gpush will tell you about it, so you won't overwrite the Changes accidentally):

$ git gpick I9ce6d9e7a3

pull/rebase your local series:

$ git gpull

Answering Criticism

Shouldn't this be implemented by the server instead of client-side scripts?

Some of the problems could be addressed on the server side, but others are inherently client-side:

• The automatic "downbasing" to keep a Change's base constant even after it has been rebased locally could be done on the server, indeed.
• The management of series could - in principle - be done on the server. However:
  • This requires additional -o options which would need to be passed to git push. Nobody wants to do that by hand, so you'd end up with a script anyway.
  • This would upload unrelated Changes just to discard them immediately.
• The status report can be done by the server - in fact, it already is done, though without giving per-Change information. However, for obvious reasons this happens only after uploading the Changes, which means you lose the potential for entirely avoiding doomed uploads.
• The concurrent update overwrite protection is inherently client-side, as only the particular clone can know what was pushed from it the last time.
• The reviewer add function as such is easy enough to use via -o options, but the irc alias lookup isn't.
• Identifying commits by Change-Id must be done client-side, as otherwise git doesn't even know what to push to the server.
• git-gpick is rather obviously a client-side tool.

This changes my workflow! This is unacceptable!

If you refuse to take advantage of the automated series management gpush offers, the only necessary change in workflow would be consistently using git gpush .. (plus --rebase after conflicted pulls/rebases) instead of git push gerrit HEAD:refs/for/foo. If you are convinced that this is unacceptable, you should probably rethink your priorities.

It's a new tool! It's harder for newbies!

Pushing to Gerrit already requires installing the Change-Id-creating commit-msg hook. Just cloning another (small) repository to get access to the scripts doesn't seem terribly hard in comparison. Also, qt5's init-repository script will clone the repository by default anyway, so the only thing necessary is adding the bin directory to git's PATH. And the basic usage of gpush is unarguably easier than that of plain git.

It makes Qt different from other Gerrit-using projects!

These scrips are in no way specific to Qt. Anyone can use them with any Gerrit installation. Upstreaming will be considered at some point.

I have my own scripts/aliases!

Good for you! But switching will be well worth it, even if you have to retrain your finger memory. Promise!

FWIW, I have aliased git-gpush to ggp, git-gpick to ggcp, and git-gpull to ggl.

Advanced topics

Series Spanning Multiple Branches

It is sometimes necessary to fix bugs before adding a new feature. The bugfixes should typically target a stable branch, while the feature is for dev. gpush supports this scenario, but not optimally - Changes within the same series cannot have different attributes (like different target branches). While this would be supportable to a certain degree, the user interface gets really messy quickly. Therefore, you have essentially two options, each with its own caveats:

1. Target a single branch and move some Changes right before final approval and integration.
   • This confuses reviewers, so definitely requires additional communication on Gerrit.
   • It's better to target the stable branch first, because these Changes need to be integrated first. Otherwise you'd need to move them back and forth if they require amending.
     • You can do that just fine while working on dev locally, provided there are no merge conflicts.
   • gpush will adjust to the re-targeting and the already integrated Changes without requiring extra work or producing churn.
2. You can designate two separate series in gpush, basing the feature series on the pending bugfix series by using the --onto switch.
   • You need to push the series one after another each time, unless you use --all mode.
   • Each series' base is remembered as an actual SHA1. That means that you need to manually adjust the 2nd series' base once it conflicts with earlier versions of the 1st series.
   • Solutions for both issues are planned, but not implemented yet.

Pushing Multiple Series at Once

gpush will currently reject multiple source specifications on the command line, and it wouldn't be entirely trivial to make it work. However, this is not expected to be a significant limitation, as it is possible to organize series with --group and --exclude, and "flush" these assignments with --all.

Multiple Conflicts During gpick

It may happen that an attempt to update a series will lead to multiple conflicts, some of which need to be --force'd while others --ignore'd. The solution is to update in several steps, each time passing the relevant switch and ranges that denote only parts of the series. Unlike in gpush, in gpick this will not change the grouping of the series (the series structure present on Gerrit will be applied).

Merges

gpush works just fine with merges, but gpick does not support manipulating them. However, there is an easy workaround:

• check out the merge via the usual Gerrit download command
• sync the state with git gpick --check --all
• now you can amend the merge and gpush it as if you had created it locally