Skip to content

ADR-0016: Kernel Modules Are Signed After the Build, by a Recorded Step

Date: 2026-10-17

Status

Accepted (2026-10-18)

Supersedes the module-signing half of the first bullet under "Negative" in ADR-0009 ("CONFIG_MODULE_SIG_ALL signs the modules with a key generated for each build ..."). The debian/changelog half of that bullet was closed by a date wrapper on the bindeb-pkg path (#171). The rest of ADR-0009 stands.

Context

Issue #77 asks for packages that two builds of one tag write bit for bit alike. Two build-matrix.yml runs of mainstream/x86_64 after #171 (runs 38991220451 and 38986603555) were compared member by member:

  • the headers package, the linux-libc-dev package and the resolved configuration were identical;
  • the image package and vmlinuz differed in three places only: the DER certificate "Build time autogenerated kernel key" embedded in the kernel, the kernel's NT_GNU_BUILD_ID, which follows from it, and the appended signature of each of the 280 modules. With the signatures cut off, no module differed.

Every leg resolved CONFIG_MODULE_SIG_ALL=y and CONFIG_MODULE_SIG_KEY="certs/signing_key.pem" from its defconfig; no fragment asked for either. With that default, certs/Makefile runs openssl req -new -nodes -x509 ... -config x509.genkey during the build, so each build trusts and signs with a key of its own, which CI then discards. A build that generates its key cannot be reproduced, and a key that is not kept cannot sign anything later.

The kernel documents the way out in Documentation/kbuild/reproducible-builds.rst, section "Module signing" (Linux 7.2.10): build with CONFIG_MODULE_SIG_ALL off and the certificate given through CONFIG_SYSTEM_TRUSTED_KEYS, then sign the modules afterwards. Three upstream facts make this work for this forge:

  • certs/Makefile generates a key only while CONFIG_MODULE_SIG_KEY is certs/signing_key.pem; with CONFIG_MODULE_SIG_KEY="", extract-cert writes an empty signing_key.x509 (certs/extract-cert.c), and the kernel trusts only the certificates in CONFIG_SYSTEM_TRUSTED_KEYS;
  • scripts/Makefile.modinst signs during modules_install only when CONFIG_MODULE_SIG_ALL is set, so bindeb-pkg packages unsigned modules;
  • scripts/sign-file.c signs with CMS_NOATTR for RSA keys, so the same key over the same module writes the same signature.

Decision

  1. The configuration trusts a certificate and generates no key. kconfig/security-hardened.config sets CONFIG_MODULE_SIG=y, # CONFIG_MODULE_SIG_ALL is not set, CONFIG_MODULE_SIG_SHA512=y, CONFIG_MODULE_SIG_KEY="" and CONFIG_SYSTEM_TRUSTED_KEYS="certs/nucleus-module-signing.pem", a path in the build directory. CONFIG_MODULE_SIG_FORCE stays off, as before.

  2. build_kernel.sh supplies the key and its certificate. --module-key=<pem> names a PEM file with the private key and the certificate. Without it, the build generates an RSA key from the kernel's own certs/default_x509.genkey, as the kernel did before, uses it for this build and deletes it after signing, whether signing succeeded or not. Either way the certificate is written to certs/nucleus-module-signing.pem in the build directory before make bindeb-pkg.

  3. A recorded step signs after packaging. scripts/sign_kernel_modules.py unpacks the image package, refuses unless the key's DER certificate is inside vmlinux, the package carries at least one module, every module is an uncompressed .ko and none is signed yet; signs every module with the build's own scripts/sign-file; verifies each signature with openssl cms -verify against the certificate; rewrites the modules' lines of DEBIAN/md5sums; and rebuilds the package with dpkg-deb --root-owner-group and the data member's compressor. It requires SOURCE_DATE_EPOCH, which clamps the times of the files it touched. The unsigned package is kept in <work-dir>/<stream>-<arch>/unsigned/, and build.json records the step under module_signing: the key's origin (supplied or ephemeral), the certificate's SHA-256, the hash, the number of modules, and the SHA-256 of the unsigned and the signed package.

  4. The artifact gate refuses an unsigned module. scripts/check_kernel_artifacts.py refuses an image package that carries no module, a compressed module, or a .ko that does not end in an appended PKCS#7 module signature.

  5. A stable key is the operator's act. Two builds are byte-identical only when they sign with one key. A long-lived signing key for releases is a credential of the operator's and is not created here; until one exists, every build signs with its own ephemeral key, as before, and a comparison of two builds compares the unsigned packages or passes both builds one key.

Consequences

Positive

  • The build itself is now deterministic in its module handling: the unsigned packages of two builds of one tag can be compared, and two builds given one key can be compared signed.
  • Signing is a separate step with its own record; the packages before and after it are both kept.
  • The gate proves every published module carries a signature; before, nothing checked.

Negative

  • Without --module-key, released packages still differ per build, by the key, as before.
  • The kernel's build ID still follows the trusted certificate, so two builds with different keys produce different vmlinuz files.
  • An external module (OpenZFS, DKMS) is not signed by this step; it loads by tainting the kernel, as before, since CONFIG_MODULE_SIG_FORCE is off.

Neutral

  • The hash stays SHA-512 and the key type RSA, the values every leg resolved before.

Release key and revisit trigger

publish-release.yml passes no --module-key, so every release signs its modules with a key generated for that build and deleted after it. This stays so: a stable release key is a credential the operator would create and hold, and a leaked one would let anyone sign modules the released kernels trust. Nothing needs one today.

Revisit when either holds, and bring the stable key to the operator as a decision:

  • a consumer (imago or another) needs out-of-tree or DKMS modules signed for a released kernel;
  • a byte-identical rebuild of a released tag has to be shown, not only of two builds with one key.

Compliance

  • tests/test_sign_kernel_modules.py: signs with a stand-in sign-file and real openssl and dpkg-deb; two runs with one key write identical packages; refuses a signature openssl does not verify, an already signed module, a compressed module, a package without modules, a certificate the kernel does not carry, a hash name that is not one, and a missing SOURCE_DATE_EPOCH, each leaving the package unchanged.
  • tests/test_check_kernel_artifacts.py: the gate refuses an image without modules, with a compressed module or with an unsigned module.
  • tests/test_build_kernel.py: the dry run states the key and the signing step; a --module-key that is not a file is refused before anything is written; build.json records module_signing.
  • .github/workflows/build-matrix.yml, job reproducibility: builds mainstream/x86_64 twice with one generated --module-key and fails unless scripts/verify-reproducibility.sh finds the two output directories byte-identical.
  • scripts/kconfig_survival.py, run for all twelve legs by verify-requirements.yml and build-matrix.yml, refuses a leg on which the signing values do not survive olddefconfig.

References

  • Issue #77 (reproducible Debian packages); #171 (debian/changelog date).
  • Linux 7.2.10: Documentation/kbuild/reproducible-builds.rst, certs/Makefile, certs/extract-cert.c, scripts/Makefile.modinst, scripts/sign-file.c, certs/default_x509.genkey.
  • ADR-0009.