Debusine aims to be an integrated solution to build, distribute and maintain a Debian-based distribution. At Debconf 25, we talked about using it to pre-test uploads to Debian unstable, and also touched on how Freexian is using it to help maintain the Debian LTS and ELTS projects.
When Debian 10 (buster) moved to ELTS status in 2024, this came with a new
difficulty that hadn’t existed for earlier releases. Debian 10 added UEFI
Secure Boot support, meaning that there are now signed variants of the boot
loader and Linux kernel packages. Debian has a
system where certain
packages are configured as needing to be signed, and those packages include
a template for a source package along with the unsigned objects themselves.
The signing service generates detached signatures for all those objects, and
then uses the template to build a source package that it uploads back to the
archive for building in the usual way.
Once buster moved to ELTS, it could no longer rely on Debian’s signing
service for all this. Freexian operates parallel infrastructure for the
archive, and now needed to operate a parallel signing service as well. By
early 2024 we were already planning to move ELTS infrastructure towards
Debusine, and so it made sense to build a signing service there as well.
Separately, we were able to obtain a Microsoft signature for Freexian’s shim build, allowing us to chain this into the trust path for most deployed x86 machines.
Freexian can help other organizations running Debian derivatives through the same process, and can provide secure signing infrastructure to the standards required for UEFI Secure Boot.
Prior art
We considered both code-signing (Debian’s current implementation) and lp-signing (Ubuntu’s current implementation) as prior art. Neither was quite suitable for various reasons.
code-signingrelies on polling a configured URL for each archive to fetch a GPG-signed list of signing requests, which would have been awkward for us to set up, and it assumes that unsigned packages are sufficiently trusted for it to be able to rundpkg -xanddpkg-source -bon them outside any containment.dpkg -xhas had the occasional security vulnerability, so this seemed unwise for a service that might need to deal with signing packages for multiple customers.lp-signingis a microservice accepting authenticated requests, and is careful to avoid needing to manipulate packages itself. However, this relies on a different and incompatible mechanism for indicating that packages should be signed, which wasn’t something we wanted to introduce in ELTS.
Workers
Debusine already had an established system of external workers that run tasks under various kinds of containment. This seems like a good fit: after all, what’s a request to sign a package but a particular kind of task? But there are some problems here: workers can run essentially arbitrary code (such as build scripts in source packages), and even though that’s under containment, we don’t want to give such machines access to highly-sensitive data such as private keys.
Fortunately, we’d already introduced the idea of different kinds of workers a few months beforehand, in order to be able to run privileged “server tasks” that have direct access to the Debusine database. We built on that and added “signing workers”, which are much like external workers except that they only run signing tasks, no other types of tasks run on them, and they have access to a private database with information about the keys managed by their Debusine instance. (Django’s support for multiple databases made this quite easy to arrange: we were able to keep everything in the same codebase.)
Key management
It’s obviously bad practice to store private key material in the clear, but at the same time the signing workers are essentially oracles that will return signatures on request while ensuring that the rest of Debusine has no access to private key material, so they need to be able to get hold of it themselves. Hardware security modules (HSMs) are designed for this kind of thing, but they can be inconvenient to manage when large numbers of keys are involved.
Some keys are more valuable than others. If the signing key used for an experimental archive leaks, the harm is unlikely to be particularly serious; but if the ELTS signing key leaks, many customers will be affected. To match this, we implemented two key protection arrangements for the time being: one suitable for low-value keys encrypts the key in software with a configured key and stores the public key and ciphertext in the database, while one suitable for high-value keys stores keys as PKCS #11 URIs that can be set up manually by an instance administrator. We packaged some YubiHSM tools to make this easier for our sysadmins.
The signing worker calls back to the Debusine server to check whether a given work request is authorized to use a given signing key. All operations related to private keys also produce an audit log entry in the private signing database, so we can track down any misuse.
Tasks
Getting Debusine to do anything new usually requires figuring out how to model the operation as a task. In this case, that was complicated by wanting to run as little code as possible on the signing workers: in particular, we didn’t want to do all the complicated package manipulations there.
The approach we landed on was a chain of three tasks:
ExtractForSigningruns on a normal external worker. It takes the result of a package build and picks out the individual files from it that need to be signed, storing them as separate artifacts.Signruns on a signing worker, and (of course) makes the actual signatures, storing them as artifacts.AssembleSignedSourceruns on a normal external worker. It takes the signed artifacts and produces a source package containing them, based on the template found in the unsigned binary package.
Workflows
Of course, we don’t want people to have to create all those tasks directly
and figure out how to connect everything together for themselves, and that’s
what
workflows
are good at. The
make_signed_source
workflow does all the heavy lifting of creating the right tasks with the
right input data and making them depend on each other in the right ways,
including fanning out multiple copies of all this if there are multiple
architectures or multiple template packages involved. Since you probably
don’t want to stop at just having the signed source packages, it also kicks
off builds to produce signed binary packages.
Even this is too low-level for most people to use directly, so we wrapped it
all up in our
debian_pipeline
workflow, which just needs to be given a few options to enable signing
support (and those options can be locked down by workspace owners).
What’s next?
In most cases this work has been enough to allow ELTS to carry on issuing kernel security updates without too much disruption, which was the main goal; but there are other uses for a signing system. We included OpenPGP support from early on, which allows Debusine to sign its own builds, and we’ll soon be extending that to sign APT repositories hosted by Debusine.
The current key protection arrangements could use some work. Supporting automatically-generated software-encrypted keys and manually-generated keys in an HSM is fine as far as it goes, but it would be good to be able to have the best of both worlds by being able to automatically generate keys protected by an HSM. This needs some care, as HSMs often have quite small limits on the number of objects they can store at any one time, and the usual workaround is to export keys from the HSM “under wrap” (encrypted by a key known only to the HSM) so that they can be imported only when needed. We have a general idea of how to do this, but doing it efficiently will need care.
We’d be very interested in hearing from organizations that need this sort of thing, especially for Debian derivatives. Debusine provides lots of other features that can help you. Please get in touch with us at sales@freexian.com if any of this sounds useful to you.