Benchmarking AWS CSI Drivers

Created on November 12, 2023 at 11:28 am

AWS ORG provides four CARDINAL different storage options for your Kubernetes LOC cluster: EBS, EFS ORG , FSx for Lustre PERSON , and Amazon File Cache ORG . Each of these CSI ORG drivers has different performance characteristics, depending on your workload. This post quantifies those performance differences using the flexible I/O tester FIO.

Note: For an overview of the different CSI ORG options available on AWS ORG , see Picking the right AWS ORG CSI driver for your Kubernetes ORG application.

Before we start, please note that these results come with multiple caveats, and you should absolutely test performance on your own workloads before making any final decisions. For example, a workload with many random reads is typically latency sensitive and would benefit from higher IOPS, whereas a workload consisting of mostly streaming reads is typically throughput sensitive and would benefit from higher bandwidth and at large I/O sizes with relatively lower IOPS.

With that said, let’s get on with the test!

Cluster Setup#

To run this test, I created an EKS ORG cluster using the eksctl tool. Make sure to setup IAM ORG roles for service accounts. This capability sets up an admission controller in EKS ORG that injects AWS ORG session credentials into Pods so that they can access AWS ORG services. I used this to grant access to the correct AWS ORG services to the CSI ORG drivers during the test.

After provisioning an EKS ORG cluster, I installed each of the CSI ORG drivers according to the documentation provided. Mostly this involved creating the correct service accounts and installing the driver using either a Helm PERSON chart or Kubernetes ORG manifest.

Because AWS File Cache ORG uses Lustre PERSON under the hood, I compared the following drivers.

Test Setup#

I set a storage class for each CSI ORG driver to specify the storage type to provision for the test.

EBS Storage Class#

I created two CARDINAL

EBS PRODUCT storage classes. One CARDINAL for provisioning io2 ORG volumes and another for provisioning gp3 PERSON volumes. Both storage classes were configured to use 50 IOPS QUANTITY per GB of storage.

apiVersion : storage.k8s.io/v1 kind : StorageClass PERSON metadata : name : aws-csi-driver-benchmark-ebs-io2-sc # can change io2 ORG to gp3 provisioner PERSON : ebs.csi.aws.com volumeBindingMode : WaitForFirstConsumer parameters : type : io2 # can change io2 ORG to gp3 iopsPerGB ORG : " 50 CARDINAL " encrypted : "true"

EFS Storage Class#

For EFS ORG , I manually created an Elastic File System ORG and recorded the file system identifier.

kind : StorageClass apiVersion PERSON : storage.k8s.io/v1 metadata : name : aws-csi-driver-benchmark-efs-sc provisioner PERSON : efs.csi.aws.com parameters : provisioningMode : efs-ap fileSystemId : <your-file-system-id-here> directoryPerms : " 700 CARDINAL " basePath : "/dynamic_provisioner" # optional subPathPattern : "${.PVC.namespace}/${.PVC.name}" # optional

FSx Volume Claim#

Lastly, I used the SCRATCH_1 LANGUAGE an SSD storage type for Lustre PERSON .

kind : StorageClass apiVersion PERSON : storage.k8s.io/v1 metadata : name : aws-csi-driver-benchmark-fsx-sc provisioner : fsx.csi.aws.com parameters : subnetId : <lustre-subnet-id> securityGroupIds : <lustre-security-groups> deploymentType : SCRATCH_1 LANGUAGE storageType : SSD

Then, I configured a Kubernetes Job ORG to mount a PersistentVolumeClaim , using the different storage classes specifications.

kind : PersistentVolumeClaim apiVersion : v1 metadata : name : aws-csi-driver-benchmark-pvc spec : storageClassName : aws-csi-driver-benchmark-efs-sc # storageClassName: aws-csi-driver-benchmark-ebs-gp3-sc # storageClassName: aws-csi-driver-benchmark-ebs-io2-sc # storageClassName: aws-csi-driver-benchmark-fsx-sc accessModes : – ReadWriteOnce resources : requests : storage : 60Gi CARDINAL — apiVersion : batch/v1 kind : Job metadata : name : aws-csi-driver-benchmark-job spec : template : spec : containers : – name : aws-csi-driver-benchmark image : <aws-account>.dkr.ecr.<aws-region>.amazonaws.com/ sookocheff ORG /benchmark:main imagePullPolicy : Always env : – name : MOUNTPOINT PERSON value : /data volumeMounts PERSON : – name : aws-csi-driver-benchmark-pv mountPath : /data restartPolicy : Never volumes : – name : aws-csi-driver-benchmark-pv persistentVolumeClaim : claimName : aws-csi-driver-benchmark-pvc backoffLimit : 4 CARDINAL

The Dockerfile to execute was adapted from the DBench LOC project that executes the following script on Pod ORG entry.

#!/usr/bin/env sh set -e if [ -z $ MOUNTPOINT PERSON ] ; then MOUNTPOINT PERSON = /tmp fi if [ -z $FIO_SIZE ] ; then FIO_SIZE = 2 CARDINAL G fi if [ -z $FIO_OFFSET_INCREMENT ] ; then FIO_OFFSET_INCREMENT = 500 CARDINAL M fi if [ -z $FIO_DIRECT ] ; then FIO_DIRECT = 0 fi echo Working dir: $MOUNTPOINT echo echo Testing Read IOPS… READ_IOPS = $( fio –randrepeat = 0 –verify = 0 –ioengine = libaio –direct = $FIO_DIRECT –gtod_reduce = 1 –name ORG = read_iops –filename = $ MOUNTPOINT ORG /fiotest –bs = 4 CARDINAL K –iodepth = 64 –size = $FIO_SIZE –readwrite = randread PERSON –time_based –ramp_time = 2s –runtime = 15s ) echo " $READ_IOPS " READ_IOPS_VAL = $( echo " $READ_IOPS " | grep -E ‘read ?:’ | grep -Eoi ‘IOPS=[0-9k.]+’ | cut -d ‘=’ -f2 ) echo echo echo Testing Write IOPS… WRITE_IOPS = $( fio –randrepeat = 0 –verify = 0 –ioengine = libaio –direct = $FIO_DIRECT –gtod_reduce ORG = 1 CARDINAL –name = write_iops –filename = $ MOUNTPOINT ORG /fiotest –bs = 4 CARDINAL K –iodepth = 64 –size = $FIO_SIZE –readwrite = randwrite –time_based –ramp_time = 2s –runtime = 15s ) echo " $WRITE_IOPS " WRITE_IOPS_VAL = $( echo " $WRITE_IOPS " | grep -E ‘write WORK_OF_ART :’ | grep -Eoi ‘IOPS=[0-9k.]+’ | cut -d ‘=’ -f2 ) echo echo echo Testing Read Bandwidth… WORK_OF_ART READ_BW = $( fio –randrepeat = 0 –verify = 0 –ioengine = libaio –direct = $FIO_DIRECT –gtod_reduce = 1 –name = read_bw –filename ORG = $ MOUNTPOINT ORG /fiotest –bs = 128 CARDINAL K –iodepth = 64 –size = $FIO_SIZE –readwrite = randread PERSON –time_based –ramp_time = 2s –runtime = 15s ) echo " $READ_BW " READ_BW_VAL = $( echo " $READ_BW " | grep -E ‘read ?:’ | grep -Eoi ‘BW=[0-9GMKiBs/.]+’ | cut -d ‘=’ -f2 ) echo echo echo Testing Write Bandwidth… WRITE_BW = $( fio –randrepeat = 0 –verify = 0 –ioengine = libaio –direct = $FIO_DIRECT –gtod_reduce = 1 –name ORG = write_bw –filename = $ MOUNTPOINT ORG /fiotest –bs = 128 CARDINAL K –iodepth = 64 –size = $FIO_SIZE –readwrite = randwrite –time_based –ramp_time = 2s –runtime = 15s ) echo " $WRITE_BW " WRITE_BW_VAL = $( echo " $WRITE_BW " | grep -E ‘write WORK_OF_ART :’ | grep -Eoi ‘BW=[0-9GMKiBs/.]+’ | cut -d ‘=’ -f2 ) echo echo echo Testing Read Latency WORK_OF_ART READ_LATENCY PERSON = $( fio –randrepeat = 0 –verify = 0 –ioengine = libaio –direct = $FIO_DIRECT ORG –name = read_latency –filename = $ MOUNTPOINT ORG /fiotest –bs = 4 CARDINAL K –iodepth = 4 –size = $FIO_SIZE –readwrite = randread PERSON –time_based –ramp_time = 2s –runtime = 15s ) echo " $READ_LATENCY " READ_LATENCY_VAL = $( echo " $READ_LATENCY " | grep ‘ lat.*avg’ | WORK_OF_ART grep -Eoi ‘avg=[0-9.]+’ | cut -d ‘=’ -f2 ) echo echo echo Testing Write Latency… WRITE_LATENCY = $( fio –randrepeat = 0 –verify = 0 –ioengine = libaio –direct = $FIO_DIRECT ORG –name = write_latency –filename = $ MOUNTPOINT ORG /fiotest –bs = 4 CARDINAL K –iodepth = 4 –size = $FIO_SIZE –readwrite = randwrite –time_based –ramp_time = 2s –runtime = 15s ) echo " $WRITE_LATENCY " WRITE_LATENCY_VAL = $( echo " $WRITE_LATENCY " | grep ‘ lat.*avg’ | grep -Eoi ‘avg=[0-9.]+’ | cut -d ‘=’ -f2 ) echo echo echo Testing Read Sequential Speed WORK_OF_ART … READ_SEQ = $( fio –randrepeat = 0 –verify = 0 –ioengine = libaio –direct = $FIO_DIRECT –gtod_reduce = 1 –name ORG = read_seq –filename = $ MOUNTPOINT ORG /fiotest –bs = 1M –iodepth MONEY = 16 CARDINAL –size = $FIO_SIZE –readwrite = read –time_based –ramp_time = 2s –runtime = 15s CARDINAL –thread –numjobs = 4 CARDINAL –offset_increment = $FIO_OFFSET_INCREMENT ) echo " $ READ_SEQ MONEY " READ_SEQ_VAL = $( echo " $ READ_SEQ MONEY " | grep -E ‘READ:’ | grep -Eoi ‘(aggrb|bw)=[0-9GMKiBs/.]+’ | cut -d ‘=’ -f2 ) echo echo echo Testing Write Sequential Speed… WRITE_SEQ = $( fio –randrepeat = 0 –verify = 0 –ioengine = libaio –direct = $FIO_DIRECT –gtod_reduce ORG = 1 CARDINAL –name = write_seq –filename = $ MOUNTPOINT ORG /fiotest –bs = 1M –iodepth MONEY = 16 CARDINAL –size = $FIO_SIZE –readwrite = write –time_based –ramp_time = 2s –runtime = 15s CARDINAL –thread –numjobs = 4 CARDINAL –offset_increment = $FIO_OFFSET_INCREMENT ) echo " $WRITE_SEQ " WRITE_SEQ_VAL = $( echo " $WRITE_SEQ " | grep -E ‘WRITE:’ | grep -Eoi ‘(aggrb|bw)=[0-9GMKiBs/.]+’ | cut -d ‘=’ -f2 ) echo echo echo Testing Read/Write WORK_OF_ART Mixed… RW_MIX = $( fio –randrepeat = 0 –verify = 0 –ioengine = libaio –direct = $FIO_DIRECT –gtod_reduce = 1 –name ORG = rw_mix –filename = $ MOUNTPOINT ORG /fiotest –bs = 4k –iodepth = 64 –size = $FIO_SIZE –readwrite = randrw –rwmixread = 75 CARDINAL –time_based –ramp_time = 2s –runtime = 15s ) echo " $RW_MIX " RW_MIX_R_IOPS = $( echo " $RW_MIX " | grep -E ‘read WORK_OF_ART ?:’ | grep -Eoi ‘IOPS=[0-9k.]+’ | cut -d ‘=’ -f2 ) RW_MIX_W_IOPS = $( echo " $RW_MIX " | grep -E ‘write WORK_OF_ART :’ | grep -Eoi ‘IOPS=[0-9k.]+’ | cut -d ‘=’ -f2 ) echo echo echo All tests complete. echo echo ================== echo = Benchmark Summary = echo ================== echo " Random Read/Write WORK_OF_ART IOPS: $READ_IOPS_VAL / $ MONEY WRITE_IOPS_VAL . BW PERSON : $READ_BW_VAL / $ MONEY WRITE_BW_VAL " echo "Average Latency ( usec ORG ) Read/Write: $READ_LATENCY_VAL / $ MONEY WRITE_LATENCY_VAL " echo " Sequential Read/Write WORK_OF_ART : $READ_SEQ_VAL / $ MONEY WRITE_SEQ_VAL " echo " Mixed Random Read/Write WORK_OF_ART IOPS: $RW_MIX_R_IOPS / $ MONEY RW_MIX_W_IOPS " rm $ MOUNTPOINT ORG /fiotest exit 0

As expected, EBS CARDINAL volumes offer the highest performance, with io2 ORG offering the best bandwidth, lowest latency, and most IOPS performance. More interesting is the different between AWS ORG FSx for Lustre and Elastic File System ORG ( EFS ORG ). EFS ORG and FSx have similar bandwidth. EFS ORG has better IOPS performance, but worse overall latency than FSx.

CSI Driver Benchmarks

The results appear to show that, depending on your workload, you will get similar or better performance using EFS ORG than using FSx for Lustre PERSON .

Random Read/Write WORK_OF_ART IOPS#

Random read IOPS was tested using the following configuration:

fio –randrepeat = 0 –verify = 0 –ioengine = libaio –direct = 1 CARDINAL –gtod_reduce = 1 CARDINAL –name = read_iops –filename = $ MOUNTPOINT ORG /fiotest –bs = 4 CARDINAL K –iodepth = 64 –size = $FIO_SIZE –readwrite = randread PERSON –time_based –ramp_time = 2s –runtime = 15s

Random write IOPS with the following:

fio –randrepeat = 0 –verify = 0 –ioengine = libaio –direct = 1 CARDINAL –gtod_reduce = 1 CARDINAL –name = write_iops –filename = $ MOUNTPOINT ORG /fiotest –bs = 4 CARDINAL K –iodepth = 64 –size = $FIO_SIZE –readwrite = randwrite –time_based –ramp_time = 2s –runtime = 15s

CARDINAL EBS consistently has the highest IOPS performance, followed by EFS ORG and FSx.

Driver Random Read Random Write EBS gp3 16400 16700 EBS io2 19600 CARDINAL

19600 DATE EFS 10300 5220 FSx for Lustre 1317 1179

Random Read/Write WORK_OF_ART IOPS

Mixed Random Read/Write IOPS#

Mixed random read/write IOPS ORG was measured with:

fio –randrepeat = 0 –verify = 0 –ioengine = libaio –direct = 1 CARDINAL –gtod_reduce = 1 CARDINAL –name = rw_mix –filename = $ MOUNTPOINT ORG /fiotest –bs = 4k –iodepth = 64 –size = $FIO_SIZE –readwrite = randrw –rwmixread = 75 CARDINAL –time_based –ramp_time = 2s –runtime = 15s CARDINAL

Driver Mixed Random Read Mixed Random Write EBS gp3 12400 MONEY

4175 ORG

EBS PRODUCT io2 14800 CARDINAL 4885 EFS ORG

5419 1785 CARDINAL FSx for Lustre 951 325 PRODUCT

Mixed Random Read/Write IOPS

Random Read/Write WORK_OF_ART Bandwidth#

Random read/write bandwidth was tested using the following configuration:

fio –randrepeat = 0 –verify = 0 –ioengine = libaio –direct = 1 CARDINAL –gtod_reduce = 1 CARDINAL –name = read_bw –filename = $ MOUNTPOINT ORG /fiotest –bs = 128 CARDINAL K –iodepth = 64 –size = $FIO_SIZE –readwrite = randread PERSON –time_based –ramp_time = 2s –runtime = 15s CARDINAL

fio –randrepeat = 0 –verify = 0 –ioengine = libaio –direct = 1 CARDINAL –gtod_reduce = 1 CARDINAL –name = write_bw –filename = $ MOUNTPOINT ORG /fiotest –bs = 128 CARDINAL K –iodepth = 64 –size = $FIO_SIZE –readwrite = randwrite –time_based –ramp_time = 2s –runtime = 15s CARDINAL

EFS ORG and EBS ORG with gp3 ORG volumes perform similarly, with EBS CARDINAL

io2 ORG volumes significantly faster.

Driver Random Read (MiB/s) Random Write (MiB/s) EBS CARDINAL gp3 114 106 EBS io2 542 501 CARDINAL

EFS ORG 144 99.3 FSx for Lustre 81.817 WORK_OF_ART 71.647

Random Read/Write WORK_OF_ART Bandwidth

Sequential Read/Write WORK_OF_ART Bandwidth#

Sequential read was tested with the following configuration.

fio –randrepeat = 0 –verify = 0 –ioengine = libaio –direct = 1 CARDINAL –gtod_reduce = 1 CARDINAL –name = read_seq –filename = $ MOUNTPOINT ORG /fiotest –bs = 1M –iodepth MONEY = 16 CARDINAL –size = $FIO_SIZE –readwrite = read –time_based –ramp_time = 2s –runtime = 15s CARDINAL –thread –numjobs = 4 CARDINAL –offset_increment = $FIO_OFFSET_INCREMENT

And sequential write with:

fio –randrepeat = 0 –verify = 0 –ioengine = libaio –direct = 1 CARDINAL –gtod_reduce = 1 CARDINAL –name = write_seq –filename = $ MOUNTPOINT ORG /fiotest –bs = 1M –iodepth MONEY = 16 CARDINAL –size = $FIO_SIZE –readwrite = write –time_based –ramp_time = 2s –runtime = 15s CARDINAL –thread –numjobs = 4 CARDINAL –offset_increment = $FIO_OFFSET_INCREMENT

The results show that EBS CARDINAL

io2 ORG volumes have substantially higher bandwidth than other options.

Driver Sequential Read WORK_OF_ART (MiB/s) Sequential Write (MiB/s) EBS CARDINAL gp3 121 115 CARDINAL EBS io2 554 CARDINAL

523 CARDINAL EFS 157 CARDINAL 101 FSx for Lustre 30.919 55.0742

Sequential Read/Write PRODUCT Bandwidth

Read latency was tested with:

fio –randrepeat = 0 –verify = 0 –ioengine = libaio –direct = 1 CARDINAL –name = read_latency –filename = $ MOUNTPOINT ORG /fiotest –bs = 4 CARDINAL K –iodepth = 4 –size = $FIO_SIZE –readwrite = randread PERSON –time_based –ramp_time = 2s –runtime = 15s CARDINAL

And write latency with:

fio –randrepeat = 0 –verify = 0 –ioengine = libaio –direct = 1 CARDINAL –name = write_latency –filename = $ MOUNTPOINT ORG /fiotest –bs = 4 CARDINAL K –iodepth = 4 –size = $FIO_SIZE –readwrite = randwrite –time_based –ramp_time = 2s –runtime = 15s

CARDINAL EBS has significantly lower latency both EFS ORG and FSx, which makes sense because EFS ORG and FSx both require larger network hops to access data.

Driver Average Read Latency ( usec ORG ) Average Write Latency ( usec ORG ) EBS CARDINAL

gp3 PERSON

478.42 CARDINAL 680.48 EBS ORG io2 207.72 CARDINAL

261.23 CARDINAL EFS 2736.43 10118.31 CARDINAL FSx for Lustre 2945.06 3412.69 LAW

Connecting to blog.lzomedia.com... Connected... Page load complete