Hello.

I (or we're) am running a Kubernetes Cluster.

It's total of 11 worker nodes. each node have a 2 HDDs and mostly 2 of NVMe Storage. on Networking side, just 1 10Gbps Ethernet port is attached to each nodes.

However, it's pretty hard to determine which distributed storage solution (or way) is fit to our situation.

We've tried a various ways to achieve.

- Longhorn: Great. Simple to Setup. Sometimes it goes to Read-Only State. (because changed block couldn't synced to other replicated storage, especially Superblock)

- OpenEBS (copied version of Longhorn iSCSI version): Same as Longhorn.

- Ceph: Great. No issues at all. But really stressful when HDDs are jammed with S.M.A.R.T. Error.

How do you manage a storage cluster like ours?

Someone suggests to "don't replicate unless you need to" or "use NFS to Shard disks" but IDK it's the best way to keep the data on failure.

Thank you in advance.

Lee.