# Spark Connector, TTL on a per-element basis within the map collection type, which compaction strategy

**URL:** <https://forum.scylladb.com/t/spark-connector-ttl-on-a-per-element-basis-within-the-map-collection-type-which-compaction-strategy/2297>\
**Category:** ScyllaDB\
**Tags:** collections, ttl, compaction, spark\
**Created:** [July 14, 2024, 4:56am UTC](https://forum.scylladb.com/t/spark-connector-ttl-on-a-per-element-basis-within-the-map-collection-type-which-compaction-strategy/2297 "2024-07-14T04:56:14Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Guy](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.scylladb.com/guy/32/9_2.png) [@Guy](https://forum.scylladb.com/u/Guy)\
**Post date:** [July 14, 2024, 4:56am UTC](https://forum.scylladb.com/t/spark-connector-ttl-on-a-per-element-basis-within-the-map-collection-type-which-compaction-strategy/2297/1 "2024-07-14T04:56:14Z")

</div>

> [@](#):
>
> [**Originally from the User Slack**](https://scylladb-users.slack.com/)
> 
> **@Ritesh:** Hello ScyllaDB users!
> 
> I am currently working with Spark and the Cassandra connector to store and modify map collections in a ScyllaDB table. So far, I have successfully implemented and run the following code:
> 
> // Write the result DataFrame to a new table in ScyllaDB  
> final\_df.rdd.saveToCassandra(“mykeyspace”, “mytable”, SomeColumns(“id”, “map1” append, “map2” append))
> 
> However, I would like to specify a TTL (time-to-live) on a per-element basis within the map collection type. I am aware of the example code in the spark-cassandra-connector repository, which applies TTL at the row level:
> 
> import com.datastax.spark.connector.writer.\_  
> …  
> rdd.saveToCassandra(“test”, “tab”, writeConf = WriteConf(ttl = TTLOption.constant(100)))  
> rdd.saveToCassandra(“test”, “tab”, writeConf = WriteConf(timestamp = TimestampOption.constant(ts)))
> 
> Is there a way to specify TTL for each element in a map collection rather than at the row level?
> 
> Any help would be greatly appreciated!
> 
> Thanks!
> 
> ![avi](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/1X/b5e08becda8d9df1abff1d3a2d9ffe1126ac3fc6.jpeg) **@avi:** You could write each element in a separate saveToCassandra call perhaps
> 
> **@Ritesh:** @avi Thanks for the suggestion!
> 
> I used an older version of Spark and was able to set a TTL on the map column in ScyllaDB. However, despite setting the TTL to 200 seconds, my data does not expire as expected. I am still able to retrieve the data in query results using cqlsh command even after the specified TTL has passed.
> 
> ![avi](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/1X/b5e08becda8d9df1abff1d3a2d9ffe1126ac3fc6.jpeg) **@avi:** You can try to debug it by using the ttl() function to see what the database thinks your ttl is
> 
> **@Ritesh:** I tried using TTL() function on map type column(non-frozen) but I got below error-  
> “TTL expects an atomic column, but it is a non-frozen collection”
> 
> ![avi](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/1X/b5e08becda8d9df1abff1d3a2d9ffe1126ac3fc6.jpeg) **@avi:** What about TTL(my\_map[‘key’])?
> 
> **@Ritesh:** No such luck!
> 
> ![avi](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/1X/b5e08becda8d9df1abff1d3a2d9ffe1126ac3fc6.jpeg) **@avi:** I see. Please file an enhancement request.
> 
> **@Ritesh:** Okay  
> Hi @avi,
> 
> I have a question regarding our previous discussion. Since I am applying a TTL of 2 months to each element in the map, which compaction strategy would be most suitable for this scenario? Additionally, there might be monthly updates to the existing map if needed
> 
> ![avi](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/1X/b5e08becda8d9df1abff1d3a2d9ffe1126ac3fc6.jpeg) **@avi:** Size Tiered
> 
> **@Ritesh:** Can’t we use TWCS for this?
> 
> ![avi](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/1X/b5e08becda8d9df1abff1d3a2d9ffe1126ac3fc6.jpeg) **@avi:** TWCS is suitable for insert-only, no updates, and with the clustering key corresponding to time
> 
> **@Ritesh:** Okay, got it  
> Thanks!  
> Hi @avi!  
> Could you please help me with the following question:  
> Is there a way to temporarily pause TTL in ScyllaDB without re-inserting the same data with a new TTL?
> 
> ![avi](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/1X/b5e08becda8d9df1abff1d3a2d9ffe1126ac3fc6.jpeg) **@avi:** No (you can try moving your clock backwards, but likely many things will break)
> 
> **@Ritesh:** Okay