New ways to inject system CA certificates in Android 14

Created on November 12, 2023 at 10:20 am

A couple of weeks ago DATE I published a post about changes in Android 14 LAW that fundamentally break existing approaches to installing system-level CA GPE certificates, even with root access. This has triggered some fascinating discussion! I highly recommend a skim through the debate on Mastodon ORG and Hacker News ORG .

Since that was posted, quite a few people have talked to me about possible solutions, going beyond the previous approaches with new mechanisms that make it practical to do this in Android LOC 14+, and there are some good options here.

While direct root access to change these certificates by simply writing to a directory is indeed no longer possible, root is root, and so with a bit of work there are still some practical & effective ways to dig down into the internals of Android ORG and seize control of these certs once more.

Choose your own adventure:

If you just want to intercept an Android ORG

14 CARDINAL + device right now, stop reading this, download the latest HTTP Toolkit, connect your device to ADB ORG , click the ‘ Android ORG Device via ADB’ interception option for automatic setup, and dive into your traffic.

If you just want to know the steps to manually do system certificate injection on Android ORG 14 for yourself, jump down to How to install system CA GPE certificates in Android 14 FAC .

If you want the full background, so you can understand how & why this all works, read on:

Clearing up confusion

Before digging into this, I do want to explicitly clear up a few misunderstandings that I’ve seen repeatedly pop up from the previous article:

These changes don’t affect installation of CAs in other scenarios. As far as I’m aware, CA GPE installation for fully managed enterprise-provisioned devices and the limited user-installed (as opposed to system-level) CA certificates will continue functioning as before. If you’re not using root access to inject system-level CA GPE certificates into a rooted device or emulator, you don’t need to worry about this.

Similarly, it is still possible to soft-remove system CA GPE certificates using the existing toggle in the Settings UI.

This does affect AOSP ORG too, and so will presumably affect most alternative distributions unless they actively disable it.

Yes, Android ORG root access is still all powerful – it’s not literally true that Android ORG has made modifying certificates impossible, they’ve just blocked doing so directly. In the extreme case, you can build your own Android ORG images from scratch without these changes, and even without that, root access provides many mechanisms to change system behaviour. Nonetheless, moving from ‘write to a directory’ to ‘technically possible via largely undocumented internals’ is a meaningful problem, and any divergence from mainline Android ORG behaviour means more problems later (not least, maintaining that divergence as Android ORG keeps evolving). Removing users’ ability to directly modify system configuration reduces practical user control over their devices.

No, this is not likely to be an explicit manouever on the part of Google ORG . I think this is just thoughtless collateral damage (and yes, the primary goal of the actual change is a good thing!). Even unintentionally though, it’s not great that a key workflow for Android ORG privacy & security research has been thoughtlessly broken, nor that major friction has been created for users’ control of their own devices. To my eyes, whilst Google ORG aren’t actively blocking those use cases, they’re very much ‘not supported’ WONTFIX scenarios, and so impacts like this are sadly just not considered.

Under the hood

The debate around all this has lead me on a fascinating exploration, delving into the internals of Android ORG . To recap the problem quickly:

Until now, system-trusted CA GPE certificates lived in /system/etc/security/cacerts/ . On a standard AOSP ORG emulator, those could be modified directly with root access with minimal setup, immediately taking effect everywhere.

. On a standard AOSP ORG emulator, those could be modified directly with root access with minimal setup, immediately taking effect everywhere. In Android ORG

14 CARDINAL , system-trusted CA GPE certificates will generally live in /apex/ , and all of /apex is immutable.

, and all of is immutable. That APEX cacerts path cannot be remounted as rewritable – remounts simply fail. In fact, even if you unmount the entire path from a root shell, apps can still read your certificates just fine.

The alternative technique of mounting a tmpfs directory over the top also doesn’t work – even though this means that ls /apex/ ORG might return nothing (or anything else you like), apps will still see the same original data.

So, what’s going on here?

The key is that Android ORG apps are containerized – much like Docker ORG , Android ORG uses Linux PRODUCT namespaces to isolate each app’s view & access to the wider system they’re running on. There’s a few elements to this, but from our point of view, the interesting point is that each app has its own mount namespace which means that they see different mounts to what’s visible elsewhere.

You can test this out for yourself, on a rooted Android ORG device:

Open a root shell (e.g. adb shell , su )

, ) Run mount to see what’s mounted (from the perspective of the shell process)

to see what’s mounted (from the perspective of the shell process) Run ps -A

Find a process id for a normal running Android ORG app from the list

Run nsenter –mount=/proc/$PID/ns/mnt /bin/sh – This opens a shell inside that process’s mount namespace

– This opens a shell inside that process’s mount namespace Run mount PRODUCT again

again Note that the results are quite different!

For example, inside an app’s mount namespace, you’ll see different mounts including /dev/block/dm-{number} resources mounted on /data/misc/profiles/ref/{} and tmpfs mounts over /data/user and other directories.

Normally, this doesn’t matter. By default, mounts are created with ‘SHARED’ propagation, meaning that changes to mounts immediately within that path will be propagated between namespaces automatically (i.e. the mount namespaces are not fully isolated) so that everybody sees the same thing (on Linux PRODUCT , you can check the propagation of your mounts with findmnt -o TARGET ORG , PROPAGATION GPE ).

This is the case for / on Android ORG , and most other mounts, so for example in a root shell you can do this:

mkdir /data/test_directory – Create an empty target directory

– Create an empty target directory mount -t tmpfs tmpfs /data/test_directory – Mount a writable temporary in-memory filesystem there

– Mount a writable temporary in-memory filesystem there And then, for both your normal shell and in an app’s mount namespace (via nsenter as above) mount will show this: tmpfs on /data/tmp_dir type tmpfs ( rw, seclabel GPE , relatime PERSON ) I.e. both your ADB shell and all other processes on the device can see and use this mount, just as you’d expect

Unfortunately though, that’s not the case for APEX PRODUCT . The /apex mount is explicitly mounted with PRIVATE propagation, so that all changes to mounts inside the /apex path are never shared between processes.

That’s done by the init process which starts the OS, which then launches the Zygote process (with a new mount namespace copied from the parent, so including its own private /apex mount), which then in turn starts each app process whenever an app is launched on the device (who each in turn then copy that same private /apex mount).

This means from an ADB ORG shell’s mount namespace, given this private mount, it’s impossible to directly make changes to APEX PRODUCT mounts that will be visible to apps on the device. Since all APEX PRODUCT mounts are read-only, that means you can’t directly modify how any of that filesystem appears for your running apps – you can’t remove, modify or add anything. Uh oh.

So, for those of us trying to inject certificates, or indeed change any other APEX PRODUCT content, how do we solve this? In a perfect world, there’s a few specific things we’re looking for here, which the previous pre-14 Android ORG solution did provide:

Being able to add, modify and remove CA ORG certificates from a device

Not needing to inconveniently reboot the device or restart apps

Having those changes become visible to all apps immediately

Being able to do so quickly & scriptably, for easy certificate management and tool integrations

It turns out there’s at least two CARDINAL routes that tick those boxes:

Option 1: Bind-mounting through NSEnter EVENT

The key to this is the first ORDINAL caveat in the above paragraph: we can’t solve this from an ADB ORG ‘s shell’s mount namespace. Fortunately, using nsenter , we can run code in other namespaces! I’ve included a full script you can blindly run this for yourself below, but first ORDINAL let’s talk about the steps that make this work:

First ORDINAL , we need set up a writable directory somewhere. For easy compatibility with the existing approach, I’m doing this with a tmpfs mount over the (still present) non-APEX system cert directory: mount -t tmpfs tmpfs /system/etc/security/cacerts

Then you place the CA GPE certificates you’re interested in into this directory (e.g. you might want copy all the defaults out of the existing /apex/ CA GPE certificates directory) and set permissions & SELinux labels appropriately.

CA GPE certificates directory) and set permissions & SELinux labels appropriately. Then, use nsenter to enter the Zygote’s mount namespace, and bind mount this directory over the APEX ORG directory: nsenter –mount = /proc/ $ZYGOTE_PID /ns MONEY /mnt — \ /bin/mount –bind /system/etc/security/cacerts /apex/ ORG The Zygote process spawns each app, copying its mount namespace to do so, so this ensures all newly launched apps (everything started from now on) will use this.

Then, use nsenter to enter each already running app’s namespace, and do the same: nsenter –mount = /proc/ $APP_PID /ns/mnt — \ /bin/mount –bind /system/etc/security/cacerts /apex/ Alternatively ORG , if you don’t mind the awkward UX, you should be able to do the bind mount on init itself (PID 1 CARDINAL ) and then run stop && ORG start to soft-reboot the OS, recreating all the namespaces and propagating your changes everywhere (but personally I do mind the awkward reboot, so I’m ignoring that route entirely).

Bingo! Every app now sees this mount as intended, with the contents of your own directory replacing the contents of the Conscrypt PERSON module’s CA GPE certificates.

Actually doing this in practice takes a little more Bash scripting trickery to make it all run smoothly, like automatically running all those app remounts in parallel, managing permissions & SELinux labels, and dealing with Zygote vs Zygote64 – see the full script below for a ready-to-go demo.

Option 2 CARDINAL : Recursively remounting mountpoints

The second ORDINAL solution comes from PERSON , who published their own post exploring the topic. I’d suggest you read through that for the full details, but in short:

You can remount /apex manually, removing the PRIVATE propagation and making it writable (ironically, it seems that entirely removing private propagation does propagate everywhere)

manually, removing the PRIVATE propagation and making it writable (ironically, it seems that entirely removing private propagation does propagate everywhere) You copy out the entire contents of /apex/ elsewhere

elsewhere Then you unmount /apex/ entirely – removing the read-only mount that immutably provides this module

entirely – removing the read-only mount that immutably provides this module Then you copy the contents back, so it lives into the /apex mount directly, where it can be modified (you need to do this quickly, as apparently you can see crashes otherwise)

mount directly, where it can be modified (you need to do this quickly, as apparently you can see crashes otherwise) This should take effect immediately, but they recommend killing system_server (restarting all apps) to get everything back into a consistent state

As above – this is a neat trick, but it’s not my work! If you have questions on that do get in touch with @g1a55er directly.

Note that for both these solutions, this is a temporary injection – the certificates only last until the next reboot. To do this more permanently, you’ll need to permanently modify the mount configuration somehow. I haven’t investigated that myself (for testing & debugging use cases, automated temporary system re-configuration is much cleaner) but if you find a good persistent technique do please get in touch and I’ll share the details here for others.

How to install system CA GPE certificates in Android 14 PRODUCT

So, putting that together, what do you need to do in practice, to actually inject your system-level CA GPE certificate in Android 14 FAC ?

First ORDINAL , copy your CA GPE certificate onto the device, e.g. with adb push $ YOUR_CERT_FILE /data ORG /local/tmp/$CERT_HASH.0 . You’ll need the certificate hash in the filename, just as you did in previous OS versions (see my implementation of this here if you’re not sure).

Then run the below, replacing $ CERTIFICATE_PATH ORG with the path on the device (e.g. /data/local/tmp/$CERT_HASH.0 ) for the cert you want to inject:

mkdir -p -m 700 CARDINAL /data/local/tmp/tmp-ca-copy cp /apex/ PERSON /cacerts/* /data/local/tmp/tmp-ca-copy/ mount -t tmpfs tmpfs /system/etc/security/cacerts mv /data/local/tmp/tmp-ca-copy/* /system/etc/security/cacerts/ mv $ CERTIFICATE_PATH ORG /system/etc/security/cacerts/ chown root:root /system/etc/security/cacerts/* chmod 644 CARDINAL /system/etc/security/cacerts/* chcon u ORG :object_r:system_file:s0 /system/etc/security/cacerts/* ZYGOTE_PID = $( pidof zygote || true ) ZYGOTE64_PID = $( pidof zygote64 || PERSON true ) for Z_PID in " $ZYGOTE_PID " " $ZYGOTE64_PID " ; do if [ -n " $Z_PID " ] ; then nsenter –mount = /proc/ $Z_PID /ns/mnt — \ /bin/mount –bind /system/etc/security/cacerts /apex/ fi done ORG APP_PIDS = $( echo " $ZYGOTE_PID $ MONEY ZYGOTE64_PID " | \ xargs -n1 ps -o ‘PID’ -P | \ grep -v PID ) for PID ORG in $APP_PIDS ; do nsenter –mount = /proc/ $PID /ns/mnt — \ /bin/mount –bind /system/etc/security/cacerts /apex/ & ORG done wait echo "System certificate injected"

The corresponding change to fully automated this in HTTP Toolkit ORG is here.

In my testing, this works out of the box on all the rooted official Android ORG

14 CARDINAL emulators, and every other test environment I’ve managed to get my hands on (if you have a case that doesn’t work, please get in touch!). With a few tweaks (see the commit above) it’s possible to build this into a single script that works out of the box for all modern Android ORG devices, down to at least Android 7. When actually running this in practice, entering and remounting certificates within every running app on a device seems to take comfortably less than a CARDINAL

second ORDINAL , so this fits nicely within the acceptable time for automated device setup time in my use cases.

This has all come together just in time, since at the time of writing Android 14 PRODUCT is on its (likely) final beta release with a full launch coming within weeks DATE . HTTP Toolkit ORG users will have automated setup for Android 14 PRODUCT ready and waiting before any of their devices even start to update.

That’s everything from me, and hopefully that resolves this for many Android ORG versions to come. A big thanks to everybody who discussed this and shared suggestions, and especially, & ORG , who popped into my Mastodon PERSON mentions with some really helpful background & suggestions.

Have thoughts, feedback or questions? Get in touch on Mastodon GPE , on Twitter ORG or directly.

Want to inspect, debug & mock HTTP(S LOC ) traffic on your Android ORG devices and test this out? Try out HTTP Toolkit – hands-free HTTP(S LOC ) interception for mobile, web browsers, backend services, Docker ORG , and more.

Connecting to Connected... Page load complete