Originally from the User Slack
@Diego_Mendes: Hi There,
I am currently investigating alternatives to replace one of our MySQL databases currently used as a metadata server.
At this moment we have tables crossing the 10TB size and we notice soon or later we might have to reconsider our current setup.
My team is investigating between refactoring our current database to use partitioning and gain a few more years or move to a NoSQL solution, due to our current use not being pure relational, we are tempted to move away from MySQL if the effort is the same.
We came up with ScyllaDB as an option, but we have 2 requirements to satisfy the new solution:
- Data Retention Policy: We have a requirement to specify how long we want to keep the data for (e.g: 2 years) and the database engine handles the deletion. Similar to AWS S3 Retention.
- Tiered Storage: We need to find a solution that can split data into separate tiers (Hot, Cold, Freeze) similar to Elastic Search Data Tier, where we can move old data to a cheaper storage solution like HDD and S3 when not used.
I searched a bit in the ScyllaDB docs but didn’t find the answer for these, on Github it seems to have some open feature requests, but doesn’t seem to be a priority for the ScyllaDB.
Am I correct to say these features are not supported nor will be in the short term?
Are there any tools for Scylla today that can handle these use cases?
@Felipe_Cardeneti_Mendes: First one is simple, TTL.
Tiered Storage is on the roadmap.
Thought there is nothing that prevents you from manually offloading data to tiered storage. ie: TTL automatically handles expiration while data is already in another media
@Diego_Mendes: For the tiered storage I was looking for something similar to elastic search where we can keep the data in the server but with a different storage like HDD or s3.
Obrigado!