Originally from the User Slack
@Terence_Liu: Hi. Is the choice of CQL and DynamoDB Alternator interface an upfront one, and one can’t switch back and forth between them?
@Felipe_Cardeneti_Mendes: You can have both working, but CQL clients will manipulate (read/write) CQL tables, and Dynamo clients will manipulate Alternator tables. There is no inter-op atm.
@Terence_Liu: I see. That makes sense. My data modeling is just one primary key that’s evenly distributed (some md5 hash). But a few wide columns that are best modeled with CQL’s UDTs. Sometimes one additional column (like a new version of some ML record) needs to be populated for all billions of keys.
This seems to work really well with the native CQL interface, where each column itself is a RocksDB “column family” (key being the primary key, and val being the column data), and UPDATE to a new column doesn’t involve reading other existing columns.
Is this a painful operation for the alternator? I understand the underlying implementation uses the CQL Map data type in one single column. Does this mean any update to any DynamoDB field is a read-write for the entire value of that column?
@Felipe_Cardeneti_Mendes: > Is this a painful operation for the alternator? I understand the underlying implementation uses the CQL Map data type in one single column. Does this mean any update to any DynamoDB field is a read-write for the entire value of that column?
Right, Alternator tables are basically serialized as a CQL table, so you can check how we do it in fact.
a DDB hashkey is your partition key, and the rest is serialized as a :attrs column as a map<text, blob>. So you CAN update individual fields for a given key as the idea of this column is to make it work just like DDB’s schemaless idea
That said, CQL supports more types, I am biased tho.
if anything interesting: https://bsky.app/profile/felipemendes.dev/post/3l4boujjebl2f
Bluesky Social: Felipe Mendes (@felipemendes.dev)
@Terence_Liu: That was an interesting experiment. How does the storage layer treat the Map data type though? Is the entire Map a single value tracked by an LSM tree (I’m not very accurate with language here because there are partitions, but I just mean RocksDB’s column family), or each key-val pair in the Map is tracked by its own LSM tree?
The documentation has these both points. The first suggests my first understanding, but the second seems to suggest my second. I am not able to find further clarification.
> • Individual collections are not indexed internally. This means that even to access a single element of a collection, the whole collection has to be read (and reading one is not paged internally).
> • While insertion operations on sets and maps never incur a read-before-write internally, some operations on lists do. Further, some list operations are not idempotent by nature (see the section on lists below for details), making their retry in case of timeout problematic. It is thus advised to prefer sets over lists when possible.
Data Types | ScyllaDB Docs
Ohh, I see, the reality is closer to the first understanding - it’s indeed a big value. But new field inserted is treated as a separate entry, and merged during reads:
(From Claude, sounds right to me)
CREATE TABLE users (
id uuid PRIMARY KEY,
preferences map<text, text>
);
-- Initial insert
INSERT INTO users (id, preferences)
VALUES (uuid(), {'theme': 'dark', 'language': 'en'});
-- Update single key
UPDATE users
SET preferences['theme'] = 'light'
WHERE id = ...;
In this case:
- The initial insert stores the entire Map as one value
- The update to ‘theme’ doesn’t read the entire Map first (explaining the second point from docs)
- Instead, it just writes the new key-value pair for ‘theme’
- During reads, the storage layer merges these changes to present the final Map state
This design explains both points from the documentation:
• Why reading any element requires reading the whole Map (it’s stored as one unit)
• Why Map updates don’t require read-before-write (individual updates are tracked separately)
@Felipe_Cardeneti_Mendes: ^ this is correct
worth to mention that if updates to the same map happen in short fashion, then they should just get merged in memtables and flushed as one to disk
but updates to infrequently updated keys will be flushed to a different sstable and merged upon read, compaction will eventually take care of pushing the lowest tiers to higher ones
so all in all, you want to avoid a situation where you have to read too many SSTables for a given map. You should also avoid very large maps, because at the end of the day they are all stored under a single cell
@Terence_Liu: Cool - this is nice, and should be quite performant. I was only used to the RocksDB’s simple compaction of entire values, but it sure sounds like one can be fancy about compaction, and accommodate maps.
And it sounds like modeling a separate field properly as a native column is the least amount of overhead - it’s a brand new column and only queried when requested.
@Felipe_Cardeneti_Mendes: sweet, have fun! 