Originally from the User Slack
@Matheus_Salvia: I’m finding that sometimes I have some consistency issues with a particular table.
the operations are as follow:
• 1. calculate data locally and build a struct
• 2. write calculated data (single row) with CL=QUORUM
• 3. read entire partition where the row was written (they are small partitions with at most 100 rows or something - rows are also light) also with CL=QUORUM
I expect to always see the newly inserted row at step 3, but I find that sometimes that’s not the case, the row is not there. Both the read and the write are made from the same client, serially (read is only fired after write has returned).
What steps can I take to help debug this?
@avi: You can try SELECT FROM mutation_fragments() on each replica to see what it did with your data
Reading mutation fragments | ScyllaDB Docs
It basically reads from cache/memtable/sstables and shows you if some node missed the update
@Matheus_Salvia: this is gonna be hard to pull off since the problem is only very sporadic and in production
I have read in a couple of places that clock sync could be a culprit. does that make sense? if nodes and/or the client’s time drifts
@avi: Check that server clocks on the clients and servers are synced
yes
@Matheus_Salvia: will do, thanks
@Matheus_Salvia: I figured this out, wrong assumption. I trusted that queries were being made with CL=QUORUM but instead they were being made with CL=ONE. If you’re experiencing the same situation I suggest you make absolutely sure that your queries have the correct CL and then try and sync your clocks.