git-sort
and the related series.conf sorting scripts depend on the python3
dbm
and pygit2
modules.
For SLE12-SP2 and SLE12-SP3, the python3-dbm package is available in the SUSE:SLE-12:Update IBS project:
https://build.suse.de/project/show/SUSE:SLE-12:Update
http://download.suse.de/ibs/SUSE:/SLE-12:/Update/standard/SUSE:SLE-12:Update.repo
For openSUSE 42.3 and Tumbleweed, the python3-dbm package is available in the normal OSS repository.
The python3-pygit2 package is available from the Kernel:tools OBS project except for Tumbleweed where it's available from the normal OSS repository.
https://build.opensuse.org/package/show/Kernel:tools/python-pygit2
merge_tool.py depends on merge
from the rcs package, available in standard
repositories.
The functions in quilt-mode.sh
are meant to be used with a modified quilt
that can use kernel-source.git's series.conf directly instead of a shadow
copy.
Packages are available in the following OBS project https://build.opensuse.org/package/show/home:benjamin_poirier:series_sort/quilt-ks Source is avaible from https://gitlab.suse.de/benjamin_poirier/quilt
quilt depends on diffstat from the package with the same name. For SLE12-SP2 and SLE12-SP3, the diffstat package is available in the SDK module. For openSUSE 42.3 and Tumbleweed, the diffstat package is available in the normal OSS repository.
The LINUX_GIT environment variable must be set to the path of a fresh Linux
kernel git clone; it will be used as a reference for upstream commit
information. Specifically, this must be a clone of
git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git or one of the
alternate URLs found on kernel.org. The user.name
and user.email
git
config variables must be set to sensible values in that clone; they will be
used to tag patches.
If you want to import patches that are not yet in mainline but that are in a subsystem maintainer's tree, that repository must be configured as an additional remote of the local repository cloned under LINUX_GIT. For example:
linux$ git remote show
net # git://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git
net-next # git://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git
origin # git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
stable # git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
For a patch file, <patch>
which is a backport of an upstream commit:
1) add the file into one of the patches.*
directories
2) make sure that it contains the correct "Git-commit" tag
3) make sure that this commit can be found in your local LINUX_GIT clone
Then run:
kernel-source$ ./scripts/git_sort/series_insert.py <patch>
This script can also be called with multiple patches at once.
Patches which are out-of-tree (not backports of upstream commits) can be added directly at a location of your choosing in the "# out-of-tree" section of the "sorted patches" section. In doubt, simply add them at the end of that section.
After the patch has been inserted in series.conf, make sure to check that the series applies and to fix any potential conflicts. Then commit and push your result.
As upstream maintainers pull from each other, the order of patches in series.conf needs to be refreshed. In that case, run:
kernel-source$ ./scripts/series_sort.py --upstream series.conf
In case of unexpected trouble, you can also move patch entries to the "# out-of-tree" section and run the above script to reset a patch's position.
The sections "Example workflow to backport a single commit" and "Example workflow to backport a series of commits using kernel-source.git" demonstrate how to use the functions in quilt-mode.sh, which can assist in backporting a single commit or a series of commits directly to kernel-source.git.
For example, we want to backport f5a952c08e84 which is a fix for another commit which was already backported:
# adjust the path to `sequence-insert.py` according to your environment
ben@f1:~/local/src/kernel-source$ ./scripts/sequence-patch.sh $(./scripts/git_sort/sequence-insert.py f5a952c08e84)
[...]
ben@f1:~/local/src/kernel-source$ cd tmp/current
ben@f1:~/local/src/kernel-source/tmp/current$ . ../../scripts/git_sort/quilt-mode.sh
# Note that we are using the "-f" option of qcp since f5a952c08e84 is a
# followup to another commit; its log contains a "Fixes" tag. If that was not
# the case, we would use the "-d" and "-r" options of qcp.
ben@f1:~/local/src/kernel-source/tmp/current$ qcp -f f5a952c08e84
Info: using references "bsc#1026030 FATE#321670" from patch "patches.drivers/of-of_mdio-Add-a-whitelist-of-PHY-compatibilities.patch" which contains commit ae461131960b.
Importing patch /tmp/qcp.d82Wqi/0001-of-of_mdio-Add-marvell-88e1145-to-whitelist-of-PHY-c.patch (stored as patches/patches.drivers/of-of_mdio-Add-marvell-88e1145-to-whitelist-of-PHY-c.patch)
# Note that `q` is an alias for `quilt`. You may be using `q` a lot...
ben@f1:~/local/src/kernel-source/tmp/current$ q push
Applying patch patches/patches.drivers/of-of_mdio-Add-marvell-88e1145-to-whitelist-of-PHY-c.patch
File drivers/of/of_mdio.c is read-only; trying to patch anyway
patching file drivers/of/of_mdio.c
Applied patch patches/patches.drivers/of-of_mdio-Add-marvell-88e1145-to-whitelist-of-PHY-c.patch (needs refresh)
Now at patch patches/patches.drivers/of-of_mdio-Add-marvell-88e1145-to-whitelist-of-PHY-c.patch
ben@f1:~/local/src/kernel-source/tmp/current$ make olddefconfig
HOSTCC scripts/basic/fixdep
HOSTCC scripts/kconfig/conf.o
SHIPPED scripts/kconfig/zconf.tab.c
SHIPPED scripts/kconfig/zconf.lex.c
SHIPPED scripts/kconfig/zconf.hash.c
HOSTCC scripts/kconfig/zconf.tab.o
HOSTLD scripts/kconfig/conf
scripts/kconfig/conf --olddefconfig Kconfig
ben@f1:~/local/src/kernel-source/tmp/current$ qfmake
[...]
ben@f1:~/local/src/kernel-source/tmp/current$ ./refresh_patch.sh
Refreshed patch patches/patches.drivers/of-of_mdio-Add-marvell-88e1145-to-whitelist-of-PHY-c.patch
ben@f1:~/local/src/kernel-source/tmp/current$ cd ../../
ben@f1:~/local/src/kernel-source$ git st
On branch SLE12-SP3
Your branch is up-to-date with 'kerncvs/SLE12-SP3'.
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git checkout -- <file>..." to discard changes in working directory)
modified: series.conf
Untracked files:
(use "git add <file>..." to include in what will be committed)
patches.drivers/of-of_mdio-Add-marvell-88e1145-to-whitelist-of-PHY-c.patch
no changes added to commit (use "git add" and/or "git commit -a")
ben@f1:~/local/src/kernel-source$ git add -A
ben@f1:~/local/src/kernel-source$ ./scripts/log
Refer to the section "Generate the list of commit ids to backport" to generate the primary list of commits to backport, /tmp/list
Generate the work tree with patches applied up to the first patch in the list of commits to backport:
# adjust the path to `sequence-insert.py` according to your environment
kernel-source$ ./scripts/sequence-patch.sh $(./scripts/git_sort/sequence-insert.py $(head -n1 /tmp/list | awk '{print $1}'))
It is preferable to check that the driver builds before getting started:
kernel-source/tmp/current$ make -j4 drivers/net/ethernet/intel/e1000/
Import the quilt-mode functions:
kernel-source/tmp/current$ . ../../scripts/git_sort/quilt-mode.sh
Set the list of commits to backport:
kernel-source/tmp/current$ qadd -r "bsc#1024371 FATE#321245" -d patches.drivers < /tmp/list
Note that the commits are automatically sorted using git-sort.
The references and destination are saved in environment variables and reused
later by qcp
(see below). They can also be specified directly to qcp
.
The working list can be queried at any time. Note that it is kept in the $series environment variable. It will be lost if the shell exits. It is not available in other terminals.
kernel-source/tmp/current$ qnext
847a1d6796c7 e1000: Do not overestimate descriptor counts in Tx pre-check (v4.6-rc3)
kernel-source/tmp/current$ qcat
847a1d6796c7 e1000: Do not overestimate descriptor counts in Tx pre-check (v4.6-rc3)
a4605fef7132 e1000: Double Tx descriptors needed check for 82544 (v4.6-rc3)
1f2f83f83848 e1000: call ndo_stop() instead of dev_close() when running offline selftest (v4.7-rc1)
91c527a55664 ethernet/intel: use core min/max MTU checking (v4.10-rc1)
311191297125 e1000: use disable_hardirq() for e1000_netpoll() (v4.10-rc1)
Start backporting:
kernel-source/tmp/current$ qdoit -j4 drivers/net/ethernet/intel/e1000/
For each commit in the list, this command will
* go to the appropriate location in the series using qgoto
which calls
quilt push/pop
* check that the commit is not already present somewhere in the series using
qdupcheck
* import the commit using qcp
which calls git format-patch
and quilt
import
* add required tags using clean_header.sh
* apply the commit using quilt push
* build test the result using qfmake
. This calls make with the options
specified to qdoit
plus the .o targets corresponding to the .c files
changed by the topmost patch.
The process will stop automatically in case of error. At that time the user
must address the situation and then call qdoit
again when ready.
To address the situation,
* if a commit is already present in an existing patch
* possibly leave the patch where it is or move it to the current
location. To move a patch, edit series.conf. However, if the patch
is already applied, make sure to use q pop
or qgoto
first.
Then call qskip
to skip the commit.
* remove the other copy, using q delete -r <patch
, then call
qcp <commit>
and follow as indicated below (q push, qfmake,
./refresh_patch.sh)
* if a commit does not apply
q push -f # or -fm
vi-conflicts # also from git-helpers
qfmake [...]
./refresh_patch.sh
* if one or more additional commits are necessary to fix the problem
Use qedit
to add these additional commits to the list of commits to
backport.
Note that the queue of commits to backport is sorted after invoking
qadd or qedit. Therefore, commits can be added anywhere in the list
when using qedit.
After editing the queue of commits to backport, `qnext` will show one
of the new commits since it should be backported before the current
one. You can continue by calling `qdoit` to backport the dependent
commits.
q delete -r
or, if after having done q push -f
:
q pop -f
q delete -r $(q next)
The following commands can be useful to identify the origin of code lines when fixing conflicts:
quilt annotate <file>
git gui blame --line=<line> <commit> <file>
When merging or rebasing between commits in kernel-source it is possible that there is a conflict involving sorted patches in series.conf. This type of conflict can be solved automatically using the git mergetool interface with the script merge_tool.py. Please see the header of that file for installation instructions.
As an example, the merge in kernel-source commit da87d04 needed conflict resolution. Let's redo this resolution using merge_tool:
ben@f1:~/local/src/kernel-source$ git log -n1 da87d04b3b
commit da87d04b3bc6edf2b58a10e27c77352a5eb7b3d9
Merge: e2d6a02d9c 1244565fb9
Author: Jiri Kosina <jkosina@suse.cz>
Date: Wed Sep 13 18:48:33 2017 +0200
Merge remote-tracking branch 'origin/users/dchang/SLE15/for-next' into SLE15
Conflicts:
series.conf
ben@f1:~/local/src/kernel-source$ git co e2d6a02d9c
HEAD is now at e2d6a02d9c... Merge remote-tracking branch 'origin/users/bpoirier/SLE15/for-next' into SLE15
ben@f1:~/local/src/kernel-source$ git merge 1244565fb9
Auto-merging series.conf
CONFLICT (content): Merge conflict in series.conf
Recorded preimage for 'series.conf'
Automatic merge failed; fix conflicts and then commit the result.
ben@f1:~/local/src/kernel-source$ git mergetool --tool=git-sort series.conf
Merging:
series.conf
Normal merge conflict for 'series.conf':
{local}: modified file
{remote}: modified file
10 commits added, 0 commits removed from base to remote.
ben@f1:~/local/src/kernel-source$ git st
HEAD detached at e2d6a02d9c
All conflicts fixed but you are still merging.
(use "git commit" to conclude merge)
Changes to be committed:
new file: patches.drivers/be2net-Fix-UE-detection-logic-for-BE3.patch
new file: patches.drivers/be2net-Update-the-driver-version-to-11.4.0.0.patch
new file: patches.drivers/bnx2x-Remove-open-coded-carrier-check.patch
new file: patches.drivers/bnx2x-fix-format-overflow-warning.patch
new file: patches.drivers/net-broadcom-bnx2x-make-a-couple-of-const-arrays-sta.patch
new file: patches.drivers/net-phy-Make-phy_ethtool_ksettings_get-return-void.patch
new file: patches.drivers/netxen-fix-incorrect-loop-counter-decrement.patch
new file: patches.drivers/netxen-remove-writeq-readq-function-definitions.patch
new file: patches.drivers/netxen_nic-Remove-unused-pointer-hdr-in-netxen_setup.patch
new file: patches.drivers/qlge-avoid-memcpy-buffer-overflow.patch
modified: series.conf
Untracked files:
(use "git add <file>..." to include in what will be committed)
series.conf.orig
ben@f1:~/local/src/kernel-source$ git commit
The following instructions detail an older approach, before series.conf was sorted. The instructions may still be relevant to die hard users of kernel.git but the resulting series.conf will need to be reordered, which may create some context conflicts.
When the patches come from vendors and have uncertain content: run a first pass of clean_patch.sh.
patches$ for file in *; do echo $file; clean_header.sh -r "bnc#790588 FATE#313912" $file; done
Although not mandatory, this step gives an idea of what condition the patch set is in to begin with. If this step succeeds, we will be able to have nice tags at the end.
Import the patch set into kernel.git to make sure that it applies, compiles
and works. The custom SUSE tags (Patch-mainline, ...) will be lost in the
process. Before doing git am
, run armor_origin.sh which transforms the
"Git-commit" tag into a "(cherry picked from ...)" line.
patches$ for file in *; do echo $file; armor_origin.sh $file; done
kernel$ git am /tmp/patches/cleaned/*
Use git rebase -i
to add missing commits where they belong or generally fixup
what needs to be.
kernel$ git format-patch -o /tmp/patches/sp3 origin/SLE11-SP3..
As an alternative to the previous steps, use this procedure when there is no patch set that comes from the vendor and it is instead us who are doing the backport.
upstream$ git log --no-merges --topo-order --reverse --pretty=tformat:%H v3.12.6.. -- drivers/net/ethernet/emulex/benet/ > /tmp/output
Optionally, generate a description of the commits to backport.
upstream$ cat /tmp/output | xargs -n1 git log -n1 --oneline > /tmp/list
Optionally, check if commits in the list are referenced in the logs of later commits which are not in the list themselves. You may wish to review these later commits and add them to the list.
upstream$ cat /tmp/list | check_missing_fixes.sh
Optionally, check which commits in the list have already been applied to kernel-source.git. Afterwards, you may wish to regenerate the list of commit ids with a different starting point; or remove from series.conf the commits that have already been applied and cherry-pick them again during the backport; or skip them during the backport.
# note that the path is a pattern, not just a base directory
kernel-source$ cat /tmp/list | refs_in_series.sh "drivers/net/ethernet/emulex/benet/*"
kernel$ . ../kernel-source/scripts/git_sort/backport-mode.sh
# note that the pattern is quoted
kernel$ bpset -s -p ../kernel-source/ -a /tmp/list_series -c /tmp/list2 "drivers/net/ethernet/emulex/benet/*"
Examine the next commit using the following commands
bpref
bpnext
bpstat
Apply the next commit completely
bpcherry-pick-all
bpcp
Apply a subset of the next commit. The changes under the path specified to bpset are included, more can optionally be specified.
bpcherry-pick-include <path>
bpcpi
After applying a commit, you may have to fix conflicts manually. Moreover, it's a good thing to check that the result builds. Sometimes the driver commit depends on a core change. In that case, the core change can be cherry-picked and moved just before the current commit using git rebase -i.
Alternatively, instead of applying the next commit, skip it
bpskip
There is a command to automate the above steps. It applies the next commit and
checks that the result builds, for all remaining commits that were fed to
bpset
, one commit at a time. The command stops when there are problems.
After manually fixing the problems, the command can be run again to resume
where it stopped. To speed things up, make
is called with a target
directory, which is the argument.
bpdoit drivers/net/ethernet/emulex/benet/
kernel$ git format-patch -o /tmp/patches/sp3 ccdc24086d54
Check series.conf to find where the patch set will go and note the last patch before that (ex: "patches.drivers/IB-0004-mlx4-Configure-extended-active-speeds.patch") and the next patch after that (ex: "patches.drivers/iwlwifi-sp1-compatible-options").
kernel-source$ ./scripts/sequence-patch.sh patches.drivers/IB-0004-mlx4-Configure-extended-active-speeds.patch
kernel-source$ cd tmp/current
kernel-source/tmp/current$ quilt import /tmp/patches/sp3/*
kernel-source/tmp/current$ while ! ( quilt next | grep -q iwlwifi-sp1-compatible-options) &&
ksapply.sh -R "bnc#790588 FATE#313912" -s suse.com patches.drivers/; do true; done
kernel-source/tmp/current$ cd ../../
Copy the new entries to their desired location in series.conf
kernel-source/tmp/current$ vi -p series.conf tmp/current/series
kernel-source$ git add -A
kernel-source$ scripts/log