Benchmarking AWS CSI Drivers

By admin

AWS
ORG

provides

four
CARDINAL

different storage options for your

Kubernetes
LOC

cluster: EBS,

EFS
ORG

, FSx for

Lustre
PERSON

, and

Amazon File Cache
ORG

. Each of these

CSI
ORG

drivers has different performance characteristics, depending on your workload. This post quantifies those performance differences using the flexible I/O tester FIO.

Note: For an overview of the different

CSI
ORG

options available on

AWS
ORG

, see Picking the right

AWS
ORG

CSI driver for your

Kubernetes
ORG

application.

Before we start, please note that these results come with multiple caveats, and you should absolutely test performance on your own workloads before making any final decisions. For example, a workload with many random reads is typically latency sensitive and would benefit from higher IOPS, whereas a workload consisting of mostly streaming reads is typically throughput sensitive and would benefit from higher bandwidth and at large I/O sizes with relatively lower IOPS.

With that said, let’s get on with the test!

Cluster Setup#

To run this test, I created an

EKS
ORG

cluster using the eksctl tool. Make sure to setup

IAM
ORG

roles for service accounts. This capability sets up an admission controller in

EKS
ORG

that injects

AWS
ORG

session credentials into Pods so that they can access

AWS
ORG

services. I used this to grant access to the correct

AWS
ORG

services to the

CSI
ORG

drivers during the test.

After provisioning an

EKS
ORG

cluster, I installed each of the

CSI
ORG

drivers according to the documentation provided. Mostly this involved creating the correct service accounts and installing the driver using either a

Helm
PERSON

chart or

Kubernetes
ORG

manifest.

Because

AWS File Cache
ORG

uses

Lustre
PERSON

under the hood, I compared the following drivers.

Test Setup#

I set a storage class for each

CSI
ORG

driver to specify the storage type to provision for the test.

EBS Storage Class#

I created

two
CARDINAL


EBS
PRODUCT

storage classes.

One
CARDINAL

for provisioning

io2
ORG

volumes and another for provisioning

gp3
PERSON

volumes. Both storage classes were configured to use

50 IOPS
QUANTITY

per GB of storage.

apiVersion : storage.k8s.io/v1 kind :

StorageClass
PERSON

metadata : name : aws-csi-driver-benchmark-ebs-io2-sc # can change

io2
ORG

to

gp3 provisioner
PERSON

: ebs.csi.aws.com volumeBindingMode : WaitForFirstConsumer parameters : type : io2 # can change

io2
ORG

to

gp3 iopsPerGB
ORG

: "

50
CARDINAL

" encrypted : "true"

EFS Storage Class#

For

EFS
ORG

, I manually created

an Elastic File System
ORG

and recorded the file system identifier.

kind :

StorageClass apiVersion
PERSON

: storage.k8s.io/v1 metadata : name : aws-csi-driver-benchmark-efs-sc

provisioner
PERSON

: efs.csi.aws.com parameters : provisioningMode : efs-ap fileSystemId : <your-file-system-id-here> directoryPerms : "

700
CARDINAL

" basePath : "/dynamic_provisioner" # optional subPathPattern : "${.PVC.namespace}/${.PVC.name}" # optional

FSx Volume Claim#

Lastly, I used the

SCRATCH_1
LANGUAGE

an SSD storage type for

Lustre
PERSON

.

kind :

StorageClass apiVersion
PERSON

: storage.k8s.io/v1 metadata : name : aws-csi-driver-benchmark-fsx-sc provisioner : fsx.csi.aws.com parameters : subnetId : <lustre-subnet-id> securityGroupIds : <lustre-security-groups> deploymentType :

SCRATCH_1
LANGUAGE

storageType : SSD

Then, I configured a

Kubernetes Job
ORG

to mount a PersistentVolumeClaim , using the different storage classes specifications.

kind : PersistentVolumeClaim apiVersion : v1 metadata : name : aws-csi-driver-benchmark-pvc spec : storageClassName : aws-csi-driver-benchmark-efs-sc # storageClassName: aws-csi-driver-benchmark-ebs-gp3-sc # storageClassName: aws-csi-driver-benchmark-ebs-io2-sc # storageClassName: aws-csi-driver-benchmark-fsx-sc accessModes : – ReadWriteOnce resources : requests : storage :

60Gi
CARDINAL

— apiVersion : batch/v1 kind : Job metadata : name : aws-csi-driver-benchmark-job spec : template : spec : containers : – name : aws-csi-driver-benchmark image : <aws-account>.dkr.ecr.<aws-region>.amazonaws.com/

sookocheff
ORG

/benchmark:main imagePullPolicy : Always env : – name :

MOUNTPOINT
PERSON

value : /data

volumeMounts
PERSON

: – name : aws-csi-driver-benchmark-pv mountPath : /data restartPolicy : Never volumes : – name : aws-csi-driver-benchmark-pv persistentVolumeClaim : claimName : aws-csi-driver-benchmark-pvc backoffLimit :

4
CARDINAL

The Dockerfile to execute was adapted from the

DBench
LOC

project that executes the following script on

Pod
ORG

entry.

#!/usr/bin/env sh set -e if [ -z $

MOUNTPOINT
PERSON

] ; then

MOUNTPOINT
PERSON

= /tmp fi if [ -z $FIO_SIZE ] ; then FIO_SIZE =

2
CARDINAL

G fi if [ -z $FIO_OFFSET_INCREMENT ] ; then FIO_OFFSET_INCREMENT =

500
CARDINAL

M fi if [ -z $FIO_DIRECT ] ; then FIO_DIRECT = 0 fi echo Working dir: $MOUNTPOINT echo echo Testing Read IOPS… READ_IOPS = $( fio –randrepeat = 0 –verify = 0 –ioengine =

libaio –direct = $FIO_DIRECT –gtod_reduce = 1 –name
ORG

= read_iops –filename = $

MOUNTPOINT
ORG

/fiotest –bs =

4
CARDINAL

K –iodepth = 64 –size = $FIO_SIZE –readwrite =

randread
PERSON

–time_based –ramp_time = 2s –runtime = 15s ) echo " $READ_IOPS " READ_IOPS_VAL = $( echo " $READ_IOPS " | grep -E ‘read ?:’ | grep -Eoi ‘IOPS=[0-9k.]+’ | cut -d ‘=’ -f2 ) echo echo echo Testing Write IOPS… WRITE_IOPS = $( fio –randrepeat = 0 –verify = 0 –ioengine =

libaio –direct = $FIO_DIRECT –gtod_reduce
ORG

=

1
CARDINAL

–name = write_iops –filename = $

MOUNTPOINT
ORG

/fiotest –bs =

4
CARDINAL

K –iodepth = 64 –size = $FIO_SIZE –readwrite = randwrite –time_based –ramp_time = 2s –runtime = 15s ) echo " $WRITE_IOPS " WRITE_IOPS_VAL = $( echo " $WRITE_IOPS "

| grep -E ‘write
WORK_OF_ART

:’ | grep -Eoi ‘IOPS=[0-9k.]+’ | cut -d ‘=’ -f2 ) echo echo echo

Testing Read Bandwidth…
WORK_OF_ART

READ_BW = $( fio –randrepeat = 0 –verify = 0 –ioengine =

libaio –direct = $FIO_DIRECT –gtod_reduce = 1 –name = read_bw –filename
ORG

= $

MOUNTPOINT
ORG

/fiotest –bs =

128
CARDINAL

K –iodepth = 64 –size = $FIO_SIZE –readwrite =

randread
PERSON

–time_based –ramp_time = 2s –runtime = 15s ) echo " $READ_BW " READ_BW_VAL = $( echo " $READ_BW " | grep -E ‘read ?:’ | grep -Eoi ‘BW=[0-9GMKiBs/.]+’ | cut -d ‘=’ -f2 ) echo echo echo Testing Write Bandwidth… WRITE_BW = $( fio –randrepeat = 0 –verify = 0 –ioengine =

libaio –direct = $FIO_DIRECT –gtod_reduce = 1 –name
ORG

= write_bw –filename = $

MOUNTPOINT
ORG

/fiotest –bs =

128
CARDINAL

K –iodepth = 64 –size = $FIO_SIZE –readwrite = randwrite –time_based –ramp_time = 2s –runtime = 15s ) echo " $WRITE_BW " WRITE_BW_VAL = $( echo " $WRITE_BW "

| grep -E ‘write
WORK_OF_ART

:’ | grep -Eoi ‘BW=[0-9GMKiBs/.]+’ | cut -d ‘=’ -f2 ) echo echo echo

Testing Read Latency
WORK_OF_ART



READ_LATENCY
PERSON

= $( fio –randrepeat = 0 –verify = 0 –ioengine =

libaio –direct = $FIO_DIRECT
ORG

–name = read_latency –filename = $

MOUNTPOINT
ORG

/fiotest –bs =

4
CARDINAL

K –iodepth = 4 –size = $FIO_SIZE –readwrite =

randread
PERSON

–time_based –ramp_time = 2s –runtime = 15s ) echo " $READ_LATENCY " READ_LATENCY_VAL = $( echo " $READ_LATENCY "

| grep ‘ lat.*avg’ |
WORK_OF_ART

grep -Eoi ‘avg=[0-9.]+’ | cut -d ‘=’ -f2 ) echo echo echo Testing Write Latency… WRITE_LATENCY = $( fio –randrepeat = 0 –verify = 0 –ioengine =

libaio –direct = $FIO_DIRECT
ORG

–name = write_latency –filename = $

MOUNTPOINT
ORG

/fiotest –bs =

4
CARDINAL

K –iodepth = 4 –size = $FIO_SIZE –readwrite = randwrite –time_based –ramp_time = 2s –runtime = 15s ) echo " $WRITE_LATENCY " WRITE_LATENCY_VAL = $( echo " $WRITE_LATENCY " | grep ‘ lat.*avg’ | grep -Eoi ‘avg=[0-9.]+’ | cut -d ‘=’ -f2 ) echo echo echo

Testing Read Sequential Speed
WORK_OF_ART

… READ_SEQ = $( fio –randrepeat = 0 –verify = 0 –ioengine =

libaio –direct = $FIO_DIRECT –gtod_reduce = 1 –name
ORG

= read_seq –filename = $

MOUNTPOINT
ORG

/fiotest –bs =

1M –iodepth
MONEY

=

16
CARDINAL

–size = $FIO_SIZE –readwrite = read –time_based –ramp_time = 2s –runtime =

15s
CARDINAL

–thread –numjobs =

4
CARDINAL

–offset_increment = $FIO_OFFSET_INCREMENT ) echo " $

READ_SEQ
MONEY

" READ_SEQ_VAL = $( echo " $

READ_SEQ
MONEY

" | grep -E ‘READ:’ | grep -Eoi ‘(aggrb|bw)=[0-9GMKiBs/.]+’ | cut -d ‘=’ -f2 ) echo echo echo Testing Write Sequential Speed… WRITE_SEQ = $( fio –randrepeat = 0 –verify = 0 –ioengine =

libaio –direct = $FIO_DIRECT –gtod_reduce
ORG

=

1
CARDINAL

–name = write_seq –filename = $

MOUNTPOINT
ORG

/fiotest –bs =

1M –iodepth
MONEY

=

16
CARDINAL

–size = $FIO_SIZE –readwrite = write –time_based –ramp_time = 2s –runtime =

15s
CARDINAL

–thread –numjobs =

4
CARDINAL

–offset_increment = $FIO_OFFSET_INCREMENT ) echo " $WRITE_SEQ " WRITE_SEQ_VAL = $( echo " $WRITE_SEQ " | grep -E ‘WRITE:’ | grep -Eoi ‘(aggrb|bw)=[0-9GMKiBs/.]+’ | cut -d ‘=’ -f2 ) echo echo echo

Testing Read/Write
WORK_OF_ART

Mixed… RW_MIX = $( fio –randrepeat = 0 –verify = 0 –ioengine =

libaio –direct = $FIO_DIRECT –gtod_reduce = 1 –name
ORG

= rw_mix –filename = $

MOUNTPOINT
ORG

/fiotest –bs = 4k –iodepth = 64 –size = $FIO_SIZE –readwrite = randrw –rwmixread =

75
CARDINAL

–time_based –ramp_time = 2s –runtime = 15s ) echo " $RW_MIX " RW_MIX_R_IOPS = $( echo " $RW_MIX "

| grep -E ‘read
WORK_OF_ART

?:’ | grep -Eoi ‘IOPS=[0-9k.]+’ | cut -d ‘=’ -f2 ) RW_MIX_W_IOPS = $( echo " $RW_MIX "

| grep -E ‘write
WORK_OF_ART

:’ | grep -Eoi ‘IOPS=[0-9k.]+’ | cut -d ‘=’ -f2 ) echo echo echo All tests complete. echo echo ================== echo = Benchmark Summary = echo ================== echo "

Random Read/Write
WORK_OF_ART

IOPS:

$READ_IOPS_VAL / $
MONEY

WRITE_IOPS_VAL .

BW
PERSON

:

$READ_BW_VAL / $
MONEY

WRITE_BW_VAL " echo "Average Latency (

usec
ORG

) Read/Write:

$READ_LATENCY_VAL / $
MONEY

WRITE_LATENCY_VAL " echo "

Sequential Read/Write
WORK_OF_ART

:

$READ_SEQ_VAL / $
MONEY

WRITE_SEQ_VAL " echo "

Mixed Random Read/Write
WORK_OF_ART

IOPS:

$RW_MIX_R_IOPS / $
MONEY

RW_MIX_W_IOPS " rm $

MOUNTPOINT
ORG

/fiotest exit 0

As expected,

EBS
CARDINAL

volumes offer the highest performance, with

io2
ORG

offering the best bandwidth, lowest latency, and most IOPS performance. More interesting is the different between

AWS
ORG

FSx for

Lustre and Elastic File System
ORG

(

EFS
ORG

).

EFS
ORG

and FSx have similar bandwidth.

EFS
ORG

has better IOPS performance, but worse overall latency than FSx.

CSI Driver Benchmarks

The results appear to show that, depending on your workload, you will get similar or better performance using

EFS
ORG

than using FSx for

Lustre
PERSON

.


Random Read/Write
WORK_OF_ART

IOPS#

Random read IOPS was tested using the following configuration:

fio –randrepeat = 0 –verify = 0 –ioengine = libaio –direct =

1
CARDINAL

–gtod_reduce =

1
CARDINAL

–name = read_iops –filename = $

MOUNTPOINT
ORG

/fiotest –bs =

4
CARDINAL

K –iodepth = 64 –size = $FIO_SIZE –readwrite =

randread
PERSON

–time_based –ramp_time = 2s –runtime = 15s

Random write IOPS with the following:

fio –randrepeat = 0 –verify = 0 –ioengine = libaio –direct =

1
CARDINAL

–gtod_reduce =

1
CARDINAL

–name = write_iops –filename = $

MOUNTPOINT
ORG

/fiotest –bs =

4
CARDINAL

K –iodepth = 64 –size = $FIO_SIZE –readwrite = randwrite –time_based –ramp_time = 2s –runtime =

15s

CARDINAL

EBS consistently has the highest IOPS performance, followed by

EFS
ORG

and FSx.

Driver Random Read Random Write EBS gp3 16400 16700 EBS io2

19600
CARDINAL


19600
DATE

EFS 10300 5220 FSx for

Lustre 1317 1179

Random Read/Write
WORK_OF_ART

IOPS

Mixed Random Read/Write IOPS#

Mixed random read/write

IOPS
ORG

was measured with:

fio –randrepeat = 0 –verify = 0 –ioengine = libaio –direct =

1
CARDINAL

–gtod_reduce =

1
CARDINAL

–name = rw_mix –filename = $

MOUNTPOINT
ORG

/fiotest –bs = 4k –iodepth = 64 –size = $FIO_SIZE –readwrite = randrw –rwmixread =

75
CARDINAL

–time_based –ramp_time = 2s –runtime =

15s
CARDINAL

Driver Mixed Random Read Mixed Random Write EBS gp3

12400
MONEY


4175
ORG


EBS
PRODUCT

io2

14800
CARDINAL

4885

EFS
ORG


5419 1785
CARDINAL

FSx for

Lustre 951 325
PRODUCT

Mixed Random Read/Write IOPS


Random Read/Write
WORK_OF_ART

Bandwidth#

Random read/write bandwidth was tested using the following configuration:

fio –randrepeat = 0 –verify = 0 –ioengine = libaio –direct =

1
CARDINAL

–gtod_reduce =

1
CARDINAL

–name = read_bw –filename = $

MOUNTPOINT
ORG

/fiotest –bs =

128
CARDINAL

K –iodepth = 64 –size = $FIO_SIZE –readwrite =

randread
PERSON

–time_based –ramp_time = 2s –runtime =

15s
CARDINAL

fio –randrepeat = 0 –verify = 0 –ioengine = libaio –direct =

1
CARDINAL

–gtod_reduce =

1
CARDINAL

–name = write_bw –filename = $

MOUNTPOINT
ORG

/fiotest –bs =

128
CARDINAL

K –iodepth = 64 –size = $FIO_SIZE –readwrite = randwrite –time_based –ramp_time = 2s –runtime =

15s
CARDINAL


EFS
ORG

and

EBS
ORG

with

gp3
ORG

volumes perform similarly, with

EBS
CARDINAL


io2
ORG

volumes significantly faster.

Driver Random Read (MiB/s) Random Write (MiB/s)

EBS
CARDINAL

gp3 114 106 EBS io2

542 501
CARDINAL


EFS
ORG

144 99.3 FSx for

Lustre 81.817
WORK_OF_ART

71.647


Random Read/Write
WORK_OF_ART

Bandwidth


Sequential Read/Write
WORK_OF_ART

Bandwidth#

Sequential read was tested with the following configuration.

fio –randrepeat = 0 –verify = 0 –ioengine = libaio –direct =

1
CARDINAL

–gtod_reduce =

1
CARDINAL

–name = read_seq –filename = $

MOUNTPOINT
ORG

/fiotest –bs =

1M –iodepth
MONEY

=

16
CARDINAL

–size = $FIO_SIZE –readwrite = read –time_based –ramp_time = 2s –runtime =

15s
CARDINAL

–thread –numjobs =

4
CARDINAL

–offset_increment = $FIO_OFFSET_INCREMENT

And sequential write with:

fio –randrepeat = 0 –verify = 0 –ioengine = libaio –direct =

1
CARDINAL

–gtod_reduce =

1
CARDINAL

–name = write_seq –filename = $

MOUNTPOINT
ORG

/fiotest –bs =

1M –iodepth
MONEY

=

16
CARDINAL

–size = $FIO_SIZE –readwrite = write –time_based –ramp_time = 2s –runtime =

15s
CARDINAL

–thread –numjobs =

4
CARDINAL

–offset_increment = $FIO_OFFSET_INCREMENT

The results show that

EBS
CARDINAL


io2
ORG

volumes have substantially higher bandwidth than other options.


Driver Sequential Read
WORK_OF_ART

(MiB/s) Sequential Write (MiB/s)

EBS
CARDINAL

gp3 121

115
CARDINAL

EBS io2

554
CARDINAL


523
CARDINAL

EFS

157
CARDINAL

101 FSx for

Lustre 30.919 55.0742

Sequential Read/Write
PRODUCT

Bandwidth

Read latency was tested with:

fio –randrepeat = 0 –verify = 0 –ioengine = libaio –direct =

1
CARDINAL

–name = read_latency –filename = $

MOUNTPOINT
ORG

/fiotest –bs =

4
CARDINAL

K –iodepth = 4 –size = $FIO_SIZE –readwrite =

randread
PERSON

–time_based –ramp_time = 2s –runtime =

15s
CARDINAL

And write latency with:

fio –randrepeat = 0 –verify = 0 –ioengine = libaio –direct =

1
CARDINAL

–name = write_latency –filename = $

MOUNTPOINT
ORG

/fiotest –bs =

4
CARDINAL

K –iodepth = 4 –size = $FIO_SIZE –readwrite = randwrite –time_based –ramp_time = 2s –runtime =

15s

CARDINAL

EBS has significantly lower latency both

EFS
ORG

and FSx, which makes sense because

EFS
ORG

and FSx both require larger network hops to access data.

Driver Average Read Latency (

usec
ORG

) Average Write Latency (

usec
ORG

)

EBS
CARDINAL


gp3
PERSON


478.42
CARDINAL

680.48

EBS
ORG

io2

207.72
CARDINAL


261.23
CARDINAL

EFS

2736.43 10118.31
CARDINAL

FSx for

Lustre 2945.06 3412.69
LAW