# Refershing data, expiring old records when uploading batch data (daily aggregates), full table scan, TTL and Secondary Indexes

**URL:** <https://forum.scylladb.com/t/refershing-data-expiring-old-records-when-uploading-batch-data-daily-aggregates-full-table-scan-ttl-and-secondary-indexes/4514>\
**Category:** ScyllaDB\
**Tags:** workload-prioritizat, data-model, ttl\
**Created:** [February 18, 2025, 9:01am UTC](https://forum.scylladb.com/t/refershing-data-expiring-old-records-when-uploading-batch-data-daily-aggregates-full-table-scan-ttl-and-secondary-indexes/4514 "2025-02-18T09:01:36Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Guy](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.scylladb.com/guy/32/9_2.png) [@Guy](https://forum.scylladb.com/u/Guy)\
**Post date:** [February 18, 2025, 9:01am UTC](https://forum.scylladb.com/t/refershing-data-expiring-old-records-when-uploading-batch-data-daily-aggregates-full-table-scan-ttl-and-secondary-indexes/4514/1 "2025-02-18T09:01:36Z")

</div>

> [@](#):
>
> [**Originally from the User Slack**](https://scylladb-users.slack.com/)
> 
> ![Igor_Q](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/a/aa1ef4202ed55016b88e29998458d403e7e5cc53.png) **@Igor\_Q:** hello, everyone  
> i have a running scylla cluster storing some features in a table (id, field, data, updated\_at) with primary key (id, field)  
> this is great for saving realtime updates by (id, field) and querying the whole wide row by (id)  
> now what i want to do is to also upload batch data (daily aggregates). upsert by (id, field) works fine here, too, but the problem is, these data are meant to be fully refreshed, so i need some way to expire old records from the table  
> i have considered the following solutions:  
> • implement full scan as described in [https://www.scylladb.com/2017/03/28/parallel-efficient-full-table-scan-scylla/](https://www.scylladb.com/2017/03/28/parallel-efficient-full-table-scan-scylla/) with `bypass caching`. this can be used to expire records from all aggregate uploads by (field, updated\_at) and also to collect some data metrics at the same time. but i am concerned about performance penalty. there’s probably non-negligible limitation on how often i can run full scans  
> • use secondary index to filter by `field` , but essentially run full scan on all records with matching `field`  
> • set short TTL (3-6 hours) for records updated from aggregates and constantly reupload them to refresh the TTL. the downside is, if constant reuploading breaks, we lose the data from scylla  
> is there something i am missing? what would you suggest?
> 
> ![avi](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/1X/b5e08becda8d9df1abff1d3a2d9ffe1126ac3fc6.jpeg) **@avi:** Full scan is a good solution here, especially with workload prioritization pushing it to only use idle time