My team is testing scylla in id mapping situation recently. And in our situation, we want to keep the first id value, so we used LWT(IF NOT EXISTS) to do insert.
However, the throughput of our test is very low, just 3000 row/s in 2-replica keyspace, and we found that the iops have reached 10k/s in each node. I know that LWT will force to sync commit log, but it’s using too much IO.
Is that normal? How could I make it better?
The test cluster: 3 nodes(16c/32G)
Indeed ScyllaDB needs to do a disk write for each LWT operation. Normally, ScyllaDB uses periodic sync for commitlog, meaning that at the end of each sync period, all accumulated writes are flushed to disk. This is not good enought for LWT as it need stronger guarantees on the persistence of writes, therefore writes done on behalf of LWT are immediately flushed to disk. The elevated IOPS count is therefore unavoidable with LWT.
LWT performs multiple cluster write rounds - prepare, promise, accept, learn, prune - and each writes to disk. Prepare, promise and accept write in sync mode.
Thank you for your reply!
Is there any solution to decrease the IOPS caused by LWT? What if using condition batch? Could it be better if I decrease the paxos rounds?