Ceph Benchmark - Fast SSDs and network speeds in a Proxmox VE Ceph Reef cluster

Fast SSDs and network speeds in a Proxmox VE Ceph Reef cluster Current fast SSD disks provide great performance, and fast network cards are becoming more affordable. Hence, this is a good point to reevaluate how quickly different network setups for Ceph can be saturated depending on how many OSDs are present in each node.

Summary

In this paper we will present the following three key findings regarding hyper-converged Ceph setups with fast disks and high network bandwidth:

• Our benchmarks show that a 10 Gbit/s network can be easily overwhelmed. Even when only using one very fast disk the network becomes a bottleneck quickly.
• A network with a bandwidth of 25 Gbit/s can also become a bottleneck. Nevertheless, some improvements can be gained through configuration changes. Routing via FRR is preferred for a full-mesh cluster over Rapid Spanning Tree Protocol (RSTP). If no fallback is needed, a simple routed setup may also be a (less resilient) option.
• When using a 100 Gbit/s network the bottleneck in the cluster seems to finally shift away from the actual hardware and toward the Ceph client. Here we observed write speeds of up to 6000 MiB/s and read speeds of up to 7000 MiB/s for a single client. However, when using multiple clients in parallel, writing at up to 9800 MiB/s and reading at 19 500 MiB/s was possible.

Proxmox VE ZFS Benchmark with NVMe

To optimize performance in hyper-converged deployments with Proxmox VE and ZFS storage, the appropriate hardware setup is essential. This benchmark presents a possible setup and its resulting performance, with the intention of supporting Proxmox users in making better decisions. Hyper-converged setups with ZFS can be deployed with Proxmox VE, starting from a single node and growing to a cluster.

We recommend the use of enterprise-class NVMe SSDs and at least a 10-gigabit network for Proxmox VE storage replication. . And as long as CPU power and memory are sufficient, a single node can reach reasonably good performance levels.

• By default, ZFS is a combined file system and logical volume manager, with various redundancy levels. Virtual machines and containers can both share the storage.
• ZFS is 100% software-defined and fully open-source.
• ZFS employs continuous integrity checks and protection against data corruption.
• A Proxmox VE and ZFS storage can be extended with additional disks on the fly, without any downtime, to match growing workloads (expansion is limited on some vdev types)
• The Proxmox VE virtualization platform has integrated ZFS storage since the release of Proxmox VE 3.4, in 2014. Since then, it has been used on thousands of servers worldwide, which has provided us with enormous amounts of feedback and experience.

Hyper-converged infrastructure with Proxmox VE virtualization platform and integrated Ceph Storage

To optimize performance in hyper-converged deployments, with Proxmox VE and Ceph storage, the appropriate hardware setup is essential. This benchmark presents possible setups and their performance outcomes, with the intention of supporting Proxmox users in making better decisions.

Summary

Hyper-converged setups can be deployed with Proxmox VE, using a cluster that contains a minimum of three nodes, enterprise class NVMe SSDs, and a 100 gigabit network (10 gigabit network is the absolute minimum requirement and already a bottleneck). As long as CPU power and RAM are sufficient, a three node cluster can reach reasonably good levels of performance.

• Since by default Ceph uses a replication of three, data will remain available, even after losing a node, thus providing a highly available, distributed storage solution—fully software-defined and 100 % open- source.
• Although it is possible to run virtual machines/containers and Ceph on the same node, a separation makes sense for larger workloads.
• To match your need for growing workloads, a Proxmox VE and Ceph server cluster can be extended with additional nodes on the fly, without any downtime.
• The Proxmox VE virtualization platform has integrated Ceph storage, since the release of Proxmox VE 3.2, in early 2014. Since then, it has been used on thousands of servers worldwide, which has provided us with an enormous amount of feedback and experience.

Benchmark Proxmox VE Ceph Cluster Performance

To optimize performance in hyper-converged deployments with Proxmox VE and Ceph storage the appropriate hardware setup can help a lot. This benchmark presents some possible setups and their performance outcomes with the intention to support Proxmox users to make better decisions.

Hyper-converged setups with Proxmox VE can already be deployed on a minimum cluster setup of three nodes, enterprise class SATA SSDs, and with a 10 gigabit network. As long as there is enough CPU power and enough RAM, a decent performance of a three node cluster is possible.

Since by default Ceph uses a replication of three, the data is still available even after losing one node, thus providing a highly available and distributed storage solution—fully software-defined and 100 % open-source.

• Although it is possible to run virtual machines/containers and Ceph on the same node, a separation does make sense in bigger workloads.
• To match your needs for growing workloads, the Proxmox VE and Ceph server clusters can be extended on the fly with additional nodes without any downtime.
• The Proxmox VE virtualization platform integrates Ceph storage since early 2014 with the release of Proxmox VE 3.2. Since then it has been used on thousands of servers worldwide, which provided an enormous amount of feedback and experience.

Read the complete Proxmox VE Ceph benchmark document...