dragonboat performance issue

[biskit]

Environment

MBP 2020 - 16GB - Monterrey

GO

1.22.2

Dragonboat version

v4@master

My application gets a lot of read writes per second. The writes are usually small (< 512B) and reads are frequent and mostly small responses (<128B). I have a three node cluster in dev running on same MBP with different ports (replicas = 3). With one NodeHost and I guess one shard
My client app times out sending requests because responses don't come back in say under 200ms. I thought my program was the problem and profing it and looking made me look here and that's how I came to find out about checkdisk

I modified checkdisk to have my setup, take 3 nodehosts on 3 different ports and 3 different data dirs.

I ran it like this:

./checkdisk -three-nodehosts -num-of-clients 5 -num-of-shards 1

and the result was

read 0, 0 reads per second
total 2560, 42 proposals per second

if you add my application latency to it, it will obviously go lower. i believe the issue is the linearization of writes and reads... what knobs do i have to tweak here?

i'm attaching my version of checkdisk/main.go here

// Copyright 2018-2019 Lei Ni (nilei81@gmail.com) and other contributors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
//     http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package main

import (
        "flag"
        "fmt"
        "io"
        "log"
        "os"
        "runtime"
        "runtime/pprof"
        "sync/atomic"
        "time"

        "github.com/lni/goutils/syncutil"

        "github.com/lni/dragonboat/v4"
        "github.com/lni/dragonboat/v4/config"
        "github.com/lni/dragonboat/v4/internal/logdb"
        "github.com/lni/dragonboat/v4/internal/logdb/kv/pebble"
        "github.com/lni/dragonboat/v4/internal/vfs"
        "github.com/lni/dragonboat/v4/logger"
        "github.com/lni/dragonboat/v4/raftio"
        sm "github.com/lni/dragonboat/v4/statemachine"
)

const (
        dataDirectoryName  = "checkdisk-data-safe-to-delete/data1"
        dataDirectoryName2 = "checkdisk-data-safe-to-delete/data2"
        dataDirectoryName3 = "checkdisk-data-safe-to-delete/data3"
        raftAddress        = "localhost:26000"
        raftAddress2       = "localhost:26001"
        raftAddress3       = "localhost:26002"
)

var shardcount = flag.Int("num-of-shards", 48, "number of raft shards")
var read = flag.Bool("enable-read", false, "enable read")
var readonly = flag.Bool("read-only", false, "read only")
var batched = flag.Bool("batched-logdb", false, "use batched logdb")
var cpupprof = flag.Bool("cpu-profiling", false, "run CPU profiling")
var mempprof = flag.Bool("mem-profiling", false, "run mem profiling")
var inmemfs = flag.Bool("inmem-fs", false, "use in-memory filesystem")
var clientcount = flag.Int("num-of-clients", 10000, "number of clients to use")
var seconds = flag.Int("seconds-to-run", 60, "number of seconds to run")
var ckpt = flag.Int("checkpoint-interval", 0, "checkpoint interval")
var tiny = flag.Bool("tiny-memory", false, "tiny LogDB memory limit")
var twonh = flag.Bool("two-nodehosts", false, "use two nodehosts")
var threenh = flag.Bool("three-nodehosts", false, "use three nodehosts")

type batchedLogDBFactory struct{}

func (batchedLogDBFactory) Create(cfg config.NodeHostConfig, cb config.LogDBCallback,
        dirs []string, lldirs []string) (raftio.ILogDB, error) {
        return logdb.NewLogDB(cfg,
                cb, dirs, lldirs, true, false, pebble.NewKVStore)
}

func (batchedLogDBFactory) Name() string {
        return "Sharded-Pebble"
}

type dummyStateMachine struct{}

func newDummyStateMachine(shardID uint64, replicaID uint64) sm.IStateMachine {
        return &dummyStateMachine{}
}

func (s *dummyStateMachine) Lookup(query interface{}) (interface{}, error) {
        return query, nil
}

func (s *dummyStateMachine) Update(e sm.Entry) (sm.Result, error) {
        return sm.Result{Value: 1}, nil
}

func (s *dummyStateMachine) SaveSnapshot(w io.Writer,
        fc sm.ISnapshotFileCollection, done <-chan struct{}) error {
        v := make([]byte, 4)
        if _, err := w.Write(v); err != nil {
                return err
        }
        return nil
}

func (s *dummyStateMachine) RecoverFromSnapshot(r io.Reader,
        files []sm.SnapshotFile, done <-chan struct{}) error {
        v := make([]byte, 4)
        if _, err := r.Read(v); err != nil {
                return err
        }
        return nil
}

func (s *dummyStateMachine) Close() error { return nil }

func main() {
        flag.Parse()
        fs := vfs.DefaultFS
        if *inmemfs {
                log.Println("using in-memory fs")
                fs = vfs.NewMemFS()
        }
        if *mempprof {
                defer func() {
                        f, err := os.Create("mem.pprof")
                        if err != nil {
                                log.Fatal("could not create memory profile: ", err)
                        }
                        defer func() {
                                if err := f.Close(); err != nil {
                                        panic(err)
                                }
                        }()
                        runtime.GC()
                        if err := pprof.WriteHeapProfile(f); err != nil {
                                log.Fatal("could not write memory profile: ", err)
                        }
                }()
                log.Println("memory profile will be saved into file mem.pprof")
        }
        if err := fs.RemoveAll(dataDirectoryName); err != nil {
                panic(err)
        }
        if err := fs.RemoveAll(dataDirectoryName2); err != nil {
                panic(err)
        }
        defer func() {
                if err := fs.RemoveAll(dataDirectoryName); err != nil {
                        panic(err)
                }
        }()
        defer func() {
                if err := fs.RemoveAll(dataDirectoryName2); err != nil {
                        panic(err)
                }
        }()
        logger.GetLogger("raft").SetLevel(logger.WARNING)
        logger.GetLogger("rsm").SetLevel(logger.WARNING)
        logger.GetLogger("logdb").SetLevel(logger.WARNING)
        logger.GetLogger("transport").SetLevel(logger.WARNING)
        logger.GetLogger("dragonboat").SetLevel(logger.WARNING)
        lc := config.GetLargeMemLogDBConfig()
        lc.SaveBufferSize = 64 * 1024 * 1024
        lc.KVMaxWriteBufferNumber = 8
        lc.KVWriteBufferSize = 256 * 1024 * 1024
        lc.KVLevel0FileNumCompactionTrigger = 6
        nhc := config.NodeHostConfig{
                NodeHostDir:    dataDirectoryName,
                RTTMillisecond: 200,
                RaftAddress:    raftAddress,
                Expert:         config.ExpertConfig{FS: fs, LogDB: lc},
        }
        if *tiny {
                log.Println("using tiny LogDB memory limit")
                // nhc.Expert.LogDB = config.GetTinyMemLogDBConfig()
                nhc.Expert.LogDB = config.GetSmallMemLogDBConfig()
        }
        if *batched {
                log.Println("using batched logdb")
                nhc.Expert.LogDBFactory = batchedLogDBFactory{}
        }
        nh, err := dragonboat.NewNodeHost(nhc)
        if err != nil {
                panic(err)
        }
        defer nh.Close()
        var nh2 *dragonboat.NodeHost
        var nh3 *dragonboat.NodeHost
        if *twonh || *threenh {
                nhc.NodeHostDir = dataDirectoryName2
                nhc.RaftAddress = raftAddress2
                nh2, err = dragonboat.NewNodeHost(nhc)
                if err != nil {
                        panic(err)
                }
                defer nh2.Close()
        }
        if *threenh {
                nhc.NodeHostDir = dataDirectoryName3
                nhc.RaftAddress = raftAddress3
                nh3, err = dragonboat.NewNodeHost(nhc)
                if err != nil {
                        panic(err)
                }
                defer nh3.Close()
        }
        rc := config.Config{
                ShardID:         1,
                ReplicaID:       1,
                ElectionRTT:     10,
                HeartbeatRTT:    1,
                CheckQuorum:     true,
                SnapshotEntries: uint64(*ckpt),
        }
        nodes := make(map[uint64]string)
        nodes[1] = raftAddress
        if *twonh || *threenh{
                nodes[2] = raftAddress2
        }
        if *threenh{
                nodes[3] = raftAddress3
        }
        nhList := make([]*dragonboat.NodeHost, 0)
        for i := uint64(1); i <= uint64(*shardcount); i++ {
                rc.ShardID = i
                if err := nh.StartReplica(nodes, false, newDummyStateMachine, rc); err != nil {
                        panic(err)
                }
                if *twonh || *threenh{
                        rc2 := rc
                        rc2.ReplicaID = 2
                        if err := nh2.StartReplica(nodes, false, newDummyStateMachine, rc2); err != nil {
                                panic(err)
                        }
                }
                if *threenh {
                        rc3 := rc
                        rc3.ReplicaID = 3
                        if err := nh3.StartReplica(nodes, false, newDummyStateMachine, rc3); err != nil {
                                panic(err)
                        }
                }
        }
        for i := uint64(1); i <= uint64(*shardcount); i++ {
                for j := 0; j < 10000; j++ {
                        leaderID, _, ok, err := nh.GetLeaderID(i)
                        if err != nil {
                                panic(err)
                        }
                        if !*twonh && !*threenh{
                                if ok && leaderID == 1 {
                                        nhList = append(nhList, nh)
                                        break
                                }
                        } else {
                                if ok && (leaderID == 1 || leaderID == 2 || leaderID == 3) {
                                        if leaderID == 1 {
                                                nhList = append(nhList, nh)
                                        } else if leaderID == 2 {
                                                nhList = append(nhList, nh2)
                                        } else {
                                                nhList = append(nhList, nh3)
                                        }
                                        break
                                }
                        }
                        time.Sleep(time.Millisecond)
                        if j == 9999 {
                                panic("failed to elect leader")
                        }
                }
        }
        if len(nhList) != *shardcount {
                panic(fmt.Sprintf("nhList len unexpected, %d", len(nhList)))
        }
        fmt.Printf("shards are ready, will run for %d seconds\n", *seconds)
        if *cpupprof {
                f, err := os.Create("cpu.pprof")
                if err != nil {
                        log.Fatal("could not create CPU profile: ", err)
                }
                defer func() {
                        if err := f.Close(); err != nil {
                                panic(err)
                        }
                }()
                if err := pprof.StartCPUProfile(f); err != nil {
                        log.Fatal("could not start CPU profile: ", err)
                }
                defer pprof.StopCPUProfile()
                log.Println("cpu profile will be saved into file cpu.pprof")
        }

        doneCh := make(chan struct{}, 1)
        timer := time.NewTimer(time.Duration(*seconds) * time.Second)
        defer timer.Stop()
        go func() {
                <-timer.C
                close(doneCh)
        }()
        // keep proposing for 60 seconds
        stopper := syncutil.NewStopper()
        results := make([]uint64, *clientcount)
        if !*readonly {
                for i := uint64(0); i < uint64(*clientcount); i++ {
                        workerID := i
                        stopper.RunWorker(func() {
                                shardID := (workerID % uint64(*shardcount)) + 1
                                nh := nhList[shardID-1]
                                cs := nh.GetNoOPSession(shardID)
                                cmd := make([]byte, 16)
                                results[workerID] = 0
                                for {
                                        for j := 0; j < 32; j++ {
                                                rs, err := nh.Propose(cs, cmd, 4*time.Second)
                                                if err != nil {
                                                        panic(err)
                                                }
                                                v := <-rs.ResultC()
                                                if v.Completed() {
                                                        results[workerID] = results[workerID] + 1
                                                        rs.Release()
                                                }
                                        }
                                        select {
                                        case <-doneCh:
                                                return
                                        default:
                                        }
                                }
                        })
                }
        }
        reads := struct{ v uint64 }{}
        if *read || *readonly {
                for i := uint64(0); i < uint64(*clientcount); i++ {
                        workerID := i
                        stopper.RunWorker(func() {
                                shardID := (workerID % uint64(*shardcount)) + 1
                                nh := nhList[shardID-1]
                                for {
                                        for j := 0; j < 32; j++ {
                                                rs, err := nh.ReadIndex(shardID, 4*time.Second)
                                                if err != nil {
                                                        panic(err)
                                                }
                                                v := <-rs.ResultC()
                                                if v.Completed() {
                                                        atomic.AddUint64(&reads.v, 1)
                                                        if _, err := nh.ReadLocalNode(rs, nil); err != nil {
                                                                panic(err)
                                                        }
                                                        rs.Release()
                                                }
                                        }
                                        select {
                                        case <-doneCh:
                                                return
                                        default:
                                        }
                                }
                        })
                }
        }
        stopper.Stop()
        total := uint64(0)
        for _, v := range results {
                total = total + v
        }
        rv := atomic.LoadUint64(&reads.v)
        fmt.Printf("read %d, %d reads per second\n", rv, rv/uint64(*seconds))
        fmt.Printf("total %d, %d proposals per second\n", total, total/uint64(*seconds))
}

[kevburnsjr]

It took me a while to get your code to run because it imports internal dragonboat packages.
Anyway, here are the results on my machine:

Ubuntu 24.04 in WSL 2 on Windows 11
Intel Core i5-12600K w/ 32GB DDR5 @ 5600Mhz
PCIe NVME SSD (SAMSUNG 980 PRO)

> go run main.go  -three-nodehosts -num-of-clients 5 -num-of-shards 1
total 17760, 296 proposals per second

This is typical performance on WSL based on my experience. You're not doing anything wrong.
Writes scale linearly with more clients as expected.

> go run main.go  -three-nodehosts -num-of-clients 50 -num-of-shards 1
total 182400, 3040 proposals per second

> go run main.go  -three-nodehosts -num-of-clients 500 -num-of-shards 1
total 1671645, 27860 proposals per second

Running native on the host Windows OS I get higher throughput

> ./checkdisk.exe -three-nodehosts -num-of-clients 5 -num-of-shards 1
total 30560, 509 proposals per second

So basically, anything that adds latency to your disk (like WSL or old SSDs) will degrade performance.
Synchronous fsync is the bottleneck.

FYI - These are my preferred RTT config values:

• config.NodeHostConfig.RTTMillisecond: 10
• config.Config.ElectionRTT: 100
• config.Config.HeartbeatRTT: 10

[kevburnsjr]

You're seeing 8 proposals per second per client on your laptop and I'm seeing over 100 on my desktop running the same code. The SSD on your laptop might be slow or degraded. Maybe try running a benchmark.

[biskit]

thanks for your responses. the code I pasted was code straight out of https://github.com/lni/dragonboat/blob/master/tools/checkdisk/main.go with mods for three-nodehosts

however, I had to make the dragonboat code work with pebble 1.1.4 and I can supply the diffs if anyone is interested

I will modify my code to use your config settings and see how it changes anything and report here

[biskit]

with your settings for RTT I get

read 0, 0 reads per second
total 2880, 48 proposals per second

here's my disk results

[biskit]

what you say may be true, but I need my client to not get timeouts when the config looks like above. when it sends few requests millis apart half of them time out which is not what I was expecting after looking at the performance graphs...

[kevburnsjr]

Have you tried running this performance test in a production-like environment?
You may get very different results on a cluster of AWS i4i.4xlarge.

[lni]

hi, this issue has been discussed here numerous times - you have a slow SSD on mbp, it is extremely slow in the sense that it takes like forever (compared to any enterprise SSD) to complete a fsync(), I am talking about like 100x slower. your bandwidth test results above are not useful as they only report bandwidth measured when fsync() is not involved.

for any performance measurement, please use enterprise SSD, e.g. those available on AWS.

[biskit]

thanks for reply, but to my original issue, that is:

./checkdisk -three-nodehosts -num-of-clients 5 -num-of-shards 1

giving me

read 0, 0 reads per second
total 2560, 42 proposals per second

is that also a disk issue? because, when I run the test without specifying -num-of-clients that defaults to 10000, the results are pretty decent, on the same MBP

[kevburnsjr]

I tried to find another Mac OS disk speed test to recommend but I couldn't find any good ones.
Apple doesn't list model numbers or performance specs for hard drives so your SSD could be anything.

Enterprise storage hardware typically exhibits superior fsync latency performance.
Really it's not that expensive to spin up 3 nodes for an hour or two in AWS to try it out.

[biskit]

i ran it on a local machine

Linux xxx 5.15.0-131-generic #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

and all three nodehosts are local, so does not simulate network latency

bs@aix:~$ sudo lshw -short
H/W path           Device          Class          Description
=============================================================
                                   system         
/0                                 bus            0T76PD
/0/0                               memory         64KiB BIOS
/0/27                              memory         32GiB System Memory
/0/27/0                            memory         DIMM [empty]
/0/27/1                            memory         16GiB DIMM DDR4 Synchronous Un
/0/27/2                            memory         DIMM [empty]
/0/27/3                            memory         16GiB DIMM DDR4 Synchronous Un
/0/2e                              memory         512KiB L1 cache
/0/2f                              memory         2MiB L2 cache
/0/30                              memory         16MiB L3 cache
/0/31                              processor      Intel(R) Core(TM) i9-9900 CPU 
/0/100                             bridge         8th/9th Gen Core 8-core Deskto
/0/100/1                           bridge         6th-10th Gen Core Processor PC
/0/100/1/0                         display        GA102 [GeForce RTX 3090]
/0/100/1/0.1       card1           multimedia     GA102 High Definition Audio Co
/0/100/1/0.1/0     input10         input          HDA NVidia HDMI/DP,pcm=11
/0/100/1/0.1/1     input11         input          HDA NVidia HDMI/DP,pcm=12
/0/100/1/0.1/2     input5          input          HDA NVidia HDMI/DP,pcm=3
/0/100/1/0.1/3     input6          input          HDA NVidia HDMI/DP,pcm=7
/0/100/1/0.1/4     input7          input          HDA NVidia HDMI/DP,pcm=8
/0/100/1/0.1/5     input8          input          HDA NVidia HDMI/DP,pcm=9
/0/100/1/0.1/6     input9          input          HDA NVidia HDMI/DP,pcm=10
/0/100/1.1                         bridge         Xeon E3-1200 v5/E3-1500 v5/6th
/0/100/2                           display        CoffeeLake-S GT2 [UHD Graphics
/0/100/8                           generic        Xeon E3-1200 v5/v6 / E3-1500 v
/0/100/14                          bus            200 Series/Z370 Chipset Family
/0/100/14/0        usb1            bus            xHCI Host Controller
/0/100/14/0/4                      input          AW-ELC
/0/100/14/0/8                      communication  Bluetooth wireless interface
/0/100/14/1        usb2            bus            xHCI Host Controller
/0/100/16                          communication  200 Series PCH CSME HECI #1
/0/100/17          scsi0           storage        200 Series PCH SATA controller
/0/100/17/0.0.0    /dev/sda        disk           2TB TOSHIBA DT01ACA2
/0/100/17/0.0.0/1  /dev/sda1       volume         127MiB reserved partition
/0/100/17/0.0.0/2  /dev/sda2       volume         1862GiB EXT4 volume
/0/100/1b                          bridge         200 Series PCH PCI Express Roo
/0/100/1b/0        /dev/nvme0      storage        KXG60PNV2T04 NVMe KIOXIA 2048G
/0/100/1b/0/0      hwmon1          disk           NVMe disk
/0/100/1b/0/2      /dev/ng0n1      disk           NVMe disk
/0/100/1b/0/1      /dev/nvme0n1    disk           2048GB NVMe disk
/0/100/1b/0/1/1                    volume         1074MiB Windows FAT volume
/0/100/1b/0/1/2    /dev/nvme0n1p2  volume         2GiB EXT4 volume
/0/100/1b/0/1/3    /dev/nvme0n1p3  volume         1904GiB EFI partition
/0/100/1c                          bridge         200 Series PCH PCI Express Roo
/0/100/1c/0                        bus            ASM1142 USB 3.1 Host Controlle
/0/100/1c/0/0      usb3            bus            xHCI Host Controller
/0/100/1c/0/1      usb4            bus            xHCI Host Controller
/0/100/1c.2                        bridge         200 Series PCH PCI Express Roo
/0/100/1c.2/0      wlp5s0          network        QCA9377 802.11ac Wireless Netw
/0/100/1c.3                        bridge         200 Series PCH PCI Express Roo
/0/100/1c.3/0      enp6s0          network        Killer E2500 Gigabit Ethernet 
/0/100/1d                          bridge         200 Series PCH PCI Express Roo
/0/100/1e                          generic        200 Series/Z370 Chipset Family
/0/100/1f                          bridge         Z370 Chipset LPC/eSPI Controll
/0/100/1f/0                        system         PnP device PNP0c02
/0/100/1f/1                        system         PnP device PNP0c02
/0/100/1f/2                        communication  PnP device PNP0501
/0/100/1f/3                        system         PnP device PNP0c02
/0/100/1f/4                        system         PnP device PNP0c02
/0/100/1f/5                        generic        PnP device INT3f0d
/0/100/1f/6                        system         PnP device PNP0c02
/0/100/1f/7                        system         PnP device PNP0c02
/0/100/1f/8                        system         PnP device PNP0c02
/0/100/1f/9                        system         PnP device PNP0c02
/0/100/1f.2                        memory         Memory controller
/0/100/1f.3        card0           multimedia     200 Series PCH HD Audio
/0/100/1f.3/0      input12         input          HDA Intel PCH Front Mic
/0/100/1f.3/1      input13         input          HDA Intel PCH Rear Mic
/0/100/1f.3/2      input14         input          HDA Intel PCH Line
/0/100/1f.3/3      input15         input          HDA Intel PCH Line Out Front
/0/100/1f.3/4      input16         input          HDA Intel PCH Line Out Surroun
/0/100/1f.3/5      input17         input          HDA Intel PCH Line Out CLFE
/0/100/1f.3/6      input18         input          HDA Intel PCH Line Out Side
/0/100/1f.3/7      input19         input          HDA Intel PCH Front Headphone
/0/100/1f.3/8      input20         input          HDA Intel PCH HDMI/DP,pcm=3
/0/100/1f.3/9      input21         input          HDA Intel PCH HDMI/DP,pcm=7
/0/100/1f.3/a      input22         input          HDA Intel PCH HDMI/DP,pcm=8
/0/100/1f.3/b      input23         input          HDA Intel PCH HDMI/DP,pcm=9
/0/100/1f.3/c      input24         input          HDA Intel PCH HDMI/DP,pcm=10
/0/100/1f.4                        bus            200 Series/Z370 Chipset Family
/1                 input0          input          Sleep Button
/2                 input1          input          Power Button
/3                 input2          input          Power Button
/4                 input3          input          Video Bus
/5                 input4          input          Dell WMI hotkeys

here are my results

1

./checkdisk -num-of-clients 5 -tiny-memory -three-nodehosts -num-of-shards 1

read 0, 0 reads per second
total 48320, 805 proposals per second

2

./checkdisk -num-of-clients 50 -tiny-memory -three-nodehosts -num-of-shards 1

read 0, 0 reads per second
total 486400, 8106 proposals per second

3

./checkdisk -num-of-clients 500 -tiny-memory -three-nodehosts -num-of-shards 1

read 0, 0 reads per second
total 5392000, 89866 proposals per second

• are these respectable numbers?
• AWS EC2 instance will come close to this? would a M5d.2xlarge compare?

[biskit]

and just to put it out there: (max num-of-clients, tiny=config.GetSmallMemLogDBConfig)

./checkdisk -tiny-memory -three-nodehosts -num-of-shards 24

read 0, 0 reads per second
total 25751936, 429198 proposals per second

[biskit]

above with read enabled

1

./checkdisk -tiny-memory -three-nodehosts -num-of-shards 24 -enable-read

read 21373696, 356228 reads per second
total 25477344, 424622 proposals per second

2

./checkdisk -tiny-memory -three-nodehosts -num-of-shards 1 -num-of-clients 5 -enable-read

read 50134, 835 reads per second
total 50240, 837 proposals per second

3

./checkdisk -tiny-memory -three-nodehosts -num-of-shards 1 -num-of-clients 50 -enable-read

read 491132, 8185 reads per second
total 496000, 8266 proposals per second

4

./checkdisk -tiny-memory -three-nodehosts -num-of-shards 1 -num-of-clients 500 -enable-read

read 4031400, 67190 reads per second
total 5227360, 87122 proposals per second

[kevburnsjr]

835 proposals per second with 5 clients puts your linear throughput at 167 proposals per second which puts your latency at 6ms which is pretty damn good (imho).

[lni]

is that also a disk issue? because, when I run the test without specifying -num-of-clients that defaults to 10000, the results are pretty decent, on the same MBP

I think it is still the slow fsync() on mbp issue.

i ran it on a local machine

This machine obviously has a faster SSD in terms of fsync(), but I have the feeling that it is still not enterprise SSDs.

Please note that enterprise SSDs are extremely cheap these days, the price difference between enterprise and consumer SSDs are literally disappearing. The performance issue discussed above is not limit to dragonboat/raft, you get hit by the exact same problem when running any database/consensus stuff on those slow SSDs.

[biskit]

it's an Alienware R9 with NVMe2 disks and an inactive SSD. so it went against the NVMe2 disk which is a KXG60PNV2T04 NVMe KIOXIA 2048G

not sure what the specs are but I think its fast

i have some older machines with SSDs, like this SK Hynix Gold S31 1TB SATA Gen3 2.5" Internal SSD | Up to 560MB/S | Compact 2.5" SSD Form Factor and will give it a go and report here