Benchmarking Disk Performance With FIO: A Practical Guide for Database Administrators

• Sat, Jul 20, 2024
• 5-minute read

In my previous post about LVM, I explored how we can use Logical Volume Management to efficiently manage our disks. I briefly touched on LVM’s powerful caching feature that can significantly accelerate disk access, but left out an important piece: how to properly test and measure these performance improvements. Today, I’ll walk you through creating realistic disk benchmarks using FIO (Flexible I/O Tester) that can help you understand your storage subsystem’s true capabilities.

Why Disk Benchmarking Is Critical for Database Performance

When tuning databases like Cassandra, understanding your system’s resource limitations is essential. In my experience with large-scale deployments, disk I/O frequently emerges as the primary bottleneck, especially as data volumes grow. A properly configured storage subsystem can be the difference between a database that delivers consistent, predictable performance and one that frustrates users with erratic response times.

FIO is my go-to tool for disk benchmarking because it offers unparalleled flexibility in simulating real-world workloads. Unlike simplistic tools that only provide basic throughput numbers, FIO allows you to create complex, multi-faceted tests that mirror your actual production load patterns.

Creating Your First FIO Benchmark

FIO uses a straightforward configuration file format to define test scenarios. Let’s start by creating a basic configuration that simulates typical Cassandra I/O patterns.

The configuration uses an INI-style format, where each section (enclosed in [brackets]) represents a different workload component. There’s a special [global] section where you can define settings that apply to all jobs:

[global]
name=cassandra
bs=4k
direct=1
directory=data/
time_based=1
runtime=20
write_bw_log
write_lat_log
write_hist_log
write_iops_log
group_reporting
per_job_logs=0
iomem=mmap
norandommap

A few important parameters to note:

• bs=4k: Sets the block size to 4KB, matching Cassandra’s typical I/O size
• direct=1: Uses direct I/O, bypassing the operating system cache
• runtime=20: Runs the test for 20 seconds (use longer durations for production testing)
• The various write_*_log options generate detailed metrics for analysis

Simulating Complex Database Workloads

To accurately benchmark a Cassandra cluster’s disk subsystem, we need to model several distinct I/O patterns that occur simultaneously:

1. Client Query Reads

First, let’s simulate the random reads generated by client queries:

[readers]
rw=randread
size=256M
numjobs=32

This configuration creates 32 concurrent jobs performing random reads on 256MB files, which approximates the pattern of multiple clients querying different partitions.

2. Compaction Readers

Next, we’ll model the sequential reads performed during compaction:

[compaction-readers]
rw=read
size=1G
numjobs=4
rate=50m

This adds 4 sequential readers with a throttled rate of 50MB/s each, matching Cassandra’s throttled compaction behavior.

3. Compaction Writers

Finally, we need to represent the sequential writes from the compaction process:

[compaction-writers]
rw=write
size=1G
numjobs=2
fsync=1000

This configuration adds 2 sequential writers with periodic fsync operations that simulate how Cassandra writes compacted SSTables to disk.

Running the Benchmark and Interpreting Results

To execute your benchmark, simply run:

fio cassandra-benchmark.fio

After completion, FIO will provide comprehensive statistics for each job type, including:

• IOPS: Operations per second (higher is better)
• Bandwidth: Throughput in MB/s
• Latency: Distribution of response times (p50, p95, p99, etc.)

Here’s what a sample output might look like for the readers section:

readers: (groupid=0, jobs=32): err= 0: pid=12345: 2024-07-20 15:30:25
  read: IOPS=15.2k, BW=59.3MiB/s (62.2MB/s)(1186MiB/20001msec)
    slat (nsec): min=1420, max=8512.3k, avg=4820.28, stdev=31215.12
    clat (usec): min=102, max=82514, avg=2098.42, stdev=1525.11
     lat (usec): min=104, max=82519, avg=2103.24, stdev=1527.18
    clat percentiles (usec):
     |  1.00th=[  245],  5.00th=[  506], 10.00th=[  750], 20.00th=[ 1172],
     | 30.00th=[ 1549], 40.00th=[ 1893], 50.00th=[ 2147], 60.00th=[ 2376],
     | 70.00th=[ 2638], 80.00th=[ 2999], 90.00th=[ 3523], 95.00th=[ 4080],
     | 99.00th=[ 6062], 99.50th=[ 7046], 99.90th=[10421], 99.95th=[12780],
     | 99.99th=[22414]

This output provides a wealth of information about your disk’s performance under load. Pay particular attention to the latency percentiles—especially p95 and p99—as these often correlate closely with your database’s observed query performance.

Real-World Application: Comparing Storage Options

I recently used this exact methodology to help a client evaluate whether upgrading from standard EBS volumes to Provisioned IOPS would deliver meaningful performance improvements for their Cassandra cluster. The results showed that while sequential operations saw minimal benefit, random read performance improved by 35% at p99 latency, which directly translated to more consistent query response times for their application.

Another client was able to demonstrate that their proposed NVMe-based storage configuration could handle 3x their current workload while keeping p99 latencies under their 5ms SLA, giving them confidence in their scaling plan.

Tips for Effective Benchmarking

Based on my experience with storage benchmarking, here are some best practices to ensure you get meaningful results:

1. Run extended tests: Many performance issues only emerge after extended periods. For production validation, run tests for hours or even days.

2. Warm up your system: Always discard the first few minutes of results to allow the system to reach steady state.

3. Match your block size: Use block sizes that match your database’s I/O patterns (typically 4K for Cassandra).

4. Test multiple scenarios: Beyond your current workload, test projected future workloads to understand scaling headroom.

5. Consider mixed workloads: Testing reads and writes in isolation can miss interference effects that occur in production.

Conclusion

FIO is an incredibly powerful tool that can help you understand your storage system’s true capabilities under database workloads. By creating realistic benchmarks that mirror your actual production patterns, you can make informed decisions about storage configuration, hardware selection, and performance tuning.

In my next post, I’ll show you how to visualize FIO results using open-source tools, making it easier to identify performance patterns and communicate findings to stakeholders.

If you’re using FIO for database benchmarking, I’d love to hear about your experiences. What workload patterns have you found most useful to simulate? Share your thoughts in the comments below!

References

• FIO Official Documentation
• Cassandra Performance Best Practices
• Brendan Gregg’s FIO Visualization Tools