New ways to inject system CA certificates in Android 14

By admin

A couple of weeks ago
DATE

I published a post about changes in

Android 14
LAW

that fundamentally break existing approaches to installing system-level

CA
GPE

certificates, even with root access. This has triggered some fascinating discussion! I highly recommend a skim through the debate on

Mastodon
ORG

and

Hacker News
ORG

.

Since that was posted, quite a few people have talked to me about possible solutions, going beyond the previous approaches with new mechanisms that make it practical to do this in

Android
LOC

14+, and there are some good options here.

While direct root access to change these certificates by simply writing to a directory is indeed no longer possible, root is root, and so with a bit of work there are still some practical & effective ways to dig down into the internals of

Android
ORG

and seize control of these certs once more.

Choose your own adventure:

If you just want to intercept an

Android
ORG


14
CARDINAL

+ device right now, stop reading this, download the latest HTTP Toolkit, connect your device to

ADB
ORG

, click the ‘

Android
ORG

Device via ADB’ interception option for automatic setup, and dive into your traffic.

If you just want to know the steps to manually do system certificate injection on

Android
ORG

14 for yourself, jump down to How to install system

CA
GPE

certificates in

Android 14
FAC

.

If you want the full background, so you can understand how & why this all works, read on:

Clearing up confusion

Before digging into this, I do want to explicitly clear up a few misunderstandings that I’ve seen repeatedly pop up from the previous article:

These changes don’t affect installation of CAs in other scenarios. As far as I’m aware,

CA
GPE

installation for fully managed enterprise-provisioned devices and the limited user-installed (as opposed to system-level) CA certificates will continue functioning as before. If you’re not using root access to inject system-level

CA
GPE

certificates into a rooted device or emulator, you don’t need to worry about this.

Similarly, it is still possible to soft-remove system

CA
GPE

certificates using the existing toggle in the Settings UI.

This does affect

AOSP
ORG

too, and so will presumably affect most alternative distributions unless they actively disable it.

Yes,

Android
ORG

root access is still all powerful – it’s not literally true that

Android
ORG

has made modifying certificates impossible, they’ve just blocked doing so directly. In the extreme case, you can build your own

Android
ORG

images from scratch without these changes, and even without that, root access provides many mechanisms to change system behaviour. Nonetheless, moving from ‘write to a directory’ to ‘technically possible via largely undocumented internals’ is a meaningful problem, and any divergence from mainline

Android
ORG

behaviour means more problems later (not least, maintaining that divergence as

Android
ORG

keeps evolving). Removing users’ ability to directly modify system configuration reduces practical user control over their devices.

No, this is not likely to be an explicit manouever on the part of

Google
ORG

. I think this is just thoughtless collateral damage (and yes, the primary goal of the actual change is a good thing!). Even unintentionally though, it’s not great that a key workflow for

Android
ORG

privacy & security research has been thoughtlessly broken, nor that major friction has been created for users’ control of their own devices. To my eyes, whilst

Google
ORG

aren’t actively blocking those use cases, they’re very much ‘not supported’ WONTFIX scenarios, and so impacts like this are sadly just not considered.

Under the hood

The debate around all this has lead me on a fascinating exploration, delving into the internals of

Android
ORG

. To recap the problem quickly:

Until now, system-trusted

CA
GPE

certificates lived in /system/etc/security/cacerts/ . On a standard

AOSP
ORG

emulator, those could be modified directly with root access with minimal setup, immediately taking effect everywhere.

. On a standard

AOSP
ORG

emulator, those could be modified directly with root access with minimal setup, immediately taking effect everywhere. In

Android
ORG


14
CARDINAL

, system-trusted

CA
GPE

certificates will generally live in /apex/com.android.conscrypt/cacerts , and all of /apex is immutable.

, and all of is immutable. That APEX cacerts path cannot be remounted as rewritable – remounts simply fail. In fact, even if you unmount the entire path from a root shell, apps can still read your certificates just fine.

The alternative technique of mounting a tmpfs directory over the top also doesn’t work – even though this means that ls /apex/

com.android.conscrypt/cacerts
ORG

might return nothing (or anything else you like), apps will still see the same original data.

So, what’s going on here?

The key is that

Android
ORG

apps are containerized – much like

Docker
ORG

,

Android
ORG

uses

Linux
PRODUCT

namespaces to isolate each app’s view & access to the wider system they’re running on. There’s a few elements to this, but from our point of view, the interesting point is that each app has its own mount namespace which means that they see different mounts to what’s visible elsewhere.

You can test this out for yourself, on a rooted

Android
ORG

device:

Open a root shell (e.g. adb shell , su )

, ) Run mount to see what’s mounted (from the perspective of the shell process)

to see what’s mounted (from the perspective of the shell process) Run ps -A

Find a process id for a normal running

Android
ORG

app from the list

Run nsenter –mount=/proc/$PID/ns/mnt /bin/sh – This opens a shell inside that process’s mount namespace

– This opens a shell inside that process’s mount namespace

Run mount
PRODUCT

again

again Note that the results are quite different!

For example, inside an app’s mount namespace, you’ll see different mounts including /dev/block/dm-{number} resources mounted on /data/misc/profiles/ref/{your.app.build.id} and tmpfs mounts over /data/user and other directories.

Normally, this doesn’t matter. By default, mounts are created with ‘SHARED’ propagation, meaning that changes to mounts immediately within that path will be propagated between namespaces automatically (i.e. the mount namespaces are not fully isolated) so that everybody sees the same thing (on

Linux
PRODUCT

, you can check the propagation of your mounts with

findmnt -o TARGET
ORG

,

PROPAGATION
GPE

).

This is the case for / on

Android
ORG

, and most other mounts, so for example in a root shell you can do this:

mkdir /data/test_directory – Create an empty target directory

– Create an empty target directory mount -t tmpfs tmpfs /data/test_directory – Mount a writable temporary in-memory filesystem there

– Mount a writable temporary in-memory filesystem there And then, for both your normal shell and in an app’s mount namespace (via nsenter as above) mount will show this: tmpfs on /data/tmp_dir type tmpfs ( rw,

seclabel
GPE

,

relatime
PERSON

) I.e. both your ADB shell and all other processes on the device can see and use this mount, just as you’d expect

Unfortunately though, that’s not the case for

APEX
PRODUCT

. The /apex mount is explicitly mounted with PRIVATE propagation, so that all changes to mounts inside the /apex path are never shared between processes.

That’s done by the init process which starts the OS, which then launches the Zygote process (with a new mount namespace copied from the parent, so including its own private /apex mount), which then in turn starts each app process whenever an app is launched on the device (who each in turn then copy that same private /apex mount).

This means from an

ADB
ORG

shell’s mount namespace, given this private mount, it’s impossible to directly make changes to

APEX
PRODUCT

mounts that will be visible to apps on the device. Since all

APEX
PRODUCT

mounts are read-only, that means you can’t directly modify how any of that filesystem appears for your running apps – you can’t remove, modify or add anything. Uh oh.

So, for those of us trying to inject certificates, or indeed change any other

APEX
PRODUCT

content, how do we solve this? In a perfect world, there’s a few specific things we’re looking for here, which the previous pre-14

Android
ORG

solution did provide:

Being able to add, modify and remove

CA
ORG

certificates from a device

Not needing to inconveniently reboot the device or restart apps

Having those changes become visible to all apps immediately

Being able to do so quickly & scriptably, for easy certificate management and tool integrations

It turns out there’s

at least two
CARDINAL

routes that tick those boxes:

Option 1: Bind-mounting through

NSEnter
EVENT

The key to this is the

first
ORDINAL

caveat in the above paragraph: we can’t solve this from an

ADB
ORG

‘s shell’s mount namespace. Fortunately, using nsenter , we can run code in other namespaces! I’ve included a full script you can blindly run this for yourself below, but

first
ORDINAL

let’s talk about the steps that make this work:


First
ORDINAL

, we need set up a writable directory somewhere. For easy compatibility with the existing approach, I’m doing this with a tmpfs mount over the (still present) non-APEX system cert directory: mount -t tmpfs tmpfs /system/etc/security/cacerts

Then you place the

CA
GPE

certificates you’re interested in into this directory (e.g. you might want copy all the defaults out of the existing /apex/com.android.conscrypt/cacerts/

CA
GPE

certificates directory) and set permissions & SELinux labels appropriately.


CA
GPE

certificates directory) and set permissions & SELinux labels appropriately. Then, use nsenter to enter the Zygote’s mount namespace, and bind mount this directory over the

APEX
ORG

directory: nsenter –mount = /proc/

$ZYGOTE_PID /ns
MONEY

/mnt — \ /bin/mount –bind /system/etc/security/cacerts /apex/

com.android.conscrypt/cacerts
ORG

The Zygote process spawns each app, copying its mount namespace to do so, so this ensures all newly launched apps (everything started from now on) will use this.

Then, use nsenter to enter each already running app’s namespace, and do the same: nsenter –mount = /proc/ $APP_PID /ns/mnt — \ /bin/mount –bind /system/etc/security/cacerts /apex/

com.android.conscrypt/cacerts Alternatively
ORG

, if you don’t mind the awkward UX, you should be able to do the bind mount on init itself (PID

1
CARDINAL

) and then run stop

&&
ORG

start to soft-reboot the OS, recreating all the namespaces and propagating your changes everywhere (but personally I do mind the awkward reboot, so I’m ignoring that route entirely).

Bingo! Every app now sees this mount as intended, with the contents of your own directory replacing the contents of the

Conscrypt
PERSON

module’s

CA
GPE

certificates.

Actually doing this in practice takes a little more Bash scripting trickery to make it all run smoothly, like automatically running all those app remounts in parallel, managing permissions & SELinux labels, and dealing with Zygote vs Zygote64 – see the full script below for a ready-to-go demo.

Option

2
CARDINAL

: Recursively remounting mountpoints

The

second
ORDINAL

solution comes from

infosec.exchange/@g1a55er
PERSON

, who published their own post exploring the topic. I’d suggest you read through that for the full details, but in short:

You can remount /apex manually, removing the PRIVATE propagation and making it writable (ironically, it seems that entirely removing private propagation does propagate everywhere)

manually, removing the PRIVATE propagation and making it writable (ironically, it seems that entirely removing private propagation does propagate everywhere) You copy out the entire contents of /apex/com.android.conscrypt elsewhere

elsewhere Then you unmount /apex/com.android.conscrypt entirely – removing the read-only mount that immutably provides this module

entirely – removing the read-only mount that immutably provides this module Then you copy the contents back, so it lives into the /apex mount directly, where it can be modified (you need to do this quickly, as apparently you can see crashes otherwise)

mount directly, where it can be modified (you need to do this quickly, as apparently you can see crashes otherwise) This should take effect immediately, but they recommend killing system_server (restarting all apps) to get everything back into a consistent state

As above – this is a neat trick, but it’s not my work! If you have questions on that do get in touch with @g1a55er directly.

Note that for both these solutions, this is a temporary injection – the certificates only last until the next reboot. To do this more permanently, you’ll need to permanently modify the mount configuration somehow. I haven’t investigated that myself (for testing & debugging use cases, automated temporary system re-configuration is much cleaner) but if you find a good persistent technique do please get in touch and I’ll share the details here for others.

How to install system

CA
GPE

certificates in

Android 14
PRODUCT

So, putting that together, what do you need to do in practice, to actually inject your system-level

CA
GPE

certificate in

Android 14
FAC

?


First
ORDINAL

, copy your

CA
GPE

certificate onto the device, e.g. with adb push $

YOUR_CERT_FILE /data
ORG

/local/tmp/$CERT_HASH.0 . You’ll need the certificate hash in the filename, just as you did in previous OS versions (see my implementation of this here if you’re not sure).

Then run the below, replacing $

CERTIFICATE_PATH
ORG

with the path on the device (e.g. /data/local/tmp/$CERT_HASH.0 ) for the cert you want to inject:

mkdir -p -m

700
CARDINAL

/data/local/tmp/tmp-ca-copy cp /apex/

com.android.conscrypt
PERSON

/cacerts/* /data/local/tmp/tmp-ca-copy/ mount -t tmpfs tmpfs /system/etc/security/cacerts mv /data/local/tmp/tmp-ca-copy/* /system/etc/security/cacerts/ mv $

CERTIFICATE_PATH
ORG

/system/etc/security/cacerts/ chown root:root /system/etc/security/cacerts/* chmod

644
CARDINAL

/system/etc/security/cacerts/*

chcon u
ORG

:object_r:system_file:s0 /system/etc/security/cacerts/* ZYGOTE_PID = $( pidof zygote || true ) ZYGOTE64_PID = $( pidof

zygote64 ||
PERSON

true ) for Z_PID in " $ZYGOTE_PID " " $ZYGOTE64_PID " ; do if [ -n " $Z_PID " ] ; then nsenter –mount = /proc/ $Z_PID /ns/mnt — \ /bin/mount –bind /system/etc/security/cacerts /apex/

com.android.conscrypt/cacerts fi done
ORG

APP_PIDS = $( echo "

$ZYGOTE_PID $
MONEY

ZYGOTE64_PID " | \ xargs -n1 ps -o ‘PID’ -P | \ grep -v PID ) for

PID
ORG

in $APP_PIDS ; do nsenter –mount = /proc/ $PID /ns/mnt — \ /bin/mount –bind /system/etc/security/cacerts /apex/

com.android.conscrypt/cacerts &
ORG

done wait echo "System certificate injected"

The corresponding change to fully automated this in

HTTP Toolkit
ORG

is here.

In my testing, this works out of the box on all the rooted official

Android
ORG


14
CARDINAL

emulators, and every other test environment I’ve managed to get my hands on (if you have a case that doesn’t work, please get in touch!). With a few tweaks (see the commit above) it’s possible to build this into a single script that works out of the box for all modern

Android
ORG

devices, down to at least Android 7. When actually running this in practice, entering and remounting certificates within every running app on a device seems to take comfortably

less than a
CARDINAL


second
ORDINAL

, so this fits nicely within the acceptable time for automated device setup time in my use cases.

This has all come together just in time, since at the time of writing

Android 14
PRODUCT

is on its (likely) final beta release with a full launch coming within

weeks
DATE

.

HTTP Toolkit
ORG

users will have automated setup for

Android 14
PRODUCT

ready and waiting before any of their devices even start to update.

That’s everything from me, and hopefully that resolves this for many

Android
ORG

versions to come. A big thanks to everybody who discussed this and shared suggestions, and especially mastodon.social/@tbodt,

ioc.exchange/@tmw & infosec.exchange/@g1a55er
ORG

, who popped into my

Mastodon
PERSON

mentions with some really helpful background & suggestions.

Have thoughts, feedback or questions? Get in touch on

Mastodon
GPE

, on

Twitter
ORG

or directly.

Want to inspect, debug & mock

HTTP(S
LOC

) traffic on your

Android
ORG

devices and test this out? Try out HTTP Toolkit – hands-free

HTTP(S
LOC

) interception for mobile, web browsers, backend services,

Docker
ORG

, and more.