Hi all! Here is the situation. We initially set up a 5-node Scylla cluster using Docker, then utilized the Alternator interface to create a table. We ensured that the base table and the GSI table had only one common replica node in the cluster (for example, the base table replicas are on nodes 1, 2, and 3, while GSI table replicas are on nodes 3, 4, and 5). After that, GSI synchronous write was enabled for this table. Subsequently, we stopped the nodes where the GSI table replicas were located (for instance, nodes 3, 4, and 5). At this point, we were still able to successfully write data to the table. However, based on the logic of synchronous GSI, writing data at this stage should theoretically have been unsuccessful. So I am confused that does GSI synchronous write in Scylla version 5.2 support tables created by Alternator interface?
Hi Vincent,
There is no DynamoDB API for making a GSI synchronous, but I assume you went and changed its synchronous_updates flag via CQL. That should be fine, and should work - the “synchronous view updates” feature is part of the Scylla backend implementation, and both CQL and Alternator (DynamoDB API) frontends can use it.
The “synchronous” update option was added as a contrast to the default “asynchronous” view updates - view updates that are done in the background and the user doesn’t wait for them. With synchronous updates, we do wait for the update we send over the network, but as you noticed there is one peculiarity: If we already know that the view replica is down, we don’t wait for it to come up - instead, we write a “view hint” - a record on disk that tells us to later try writing this view update. But at this point, Scylla does nothing more, and the code thinks it did what it needed to do, and returns.
You’re right that the downside to this is that it is possible that a base-write with “synchronous view updates” can be done, and afterwards a read of the view - and the new data will be missing on the view, because although the view write got written to disk (so it’s durable), it’s not yet available. You are right that it does make sense that in this case the base write should fail instead of succeeding. I’ll open an issue about this in our bug tracker.
By the way, if all three view replicas for a row are really down, you can’t actually confirm that the synchronous write didn’t do its job - you can’t read from the view and not see the new data because the read will fail. However, it is indeed possible that the dead nodes come back to life, and before the hints are replayed (which takes time), a read of the view happens and doesn’t see the new data.
I opened an issue in Scylla’s bug tracker: Synchronous view updates may not guarante read-your-own-writes if nodes are down · Issue #16229 · scylladb/scylladb · GitHub