Originally from the User Slack
@hamonica: Hello, I have a question about partitions.
There are 100 WAS servers, and each server stores 100 data points per second in ScyllaDB. On average, each server processes 100 TPS, which means 10,000 records are stored per second, and 600,000 records are stored per minute.
I plan to use a partition key based on minute + WAS ID (min_timestamp, instance_id). Would this partition be too large and negatively impact performance?
@hamonica: um… I’d like to know the recommended number of partitions. Would it be better to split the partitions into smaller sizes rather than having larger partitions?
@Botond_Dénes: In general, the more partitions the better.
@hamonica: Is there a limit on the number of partitions that can be created in a day? Or is there a recommended number of partitions?
@Botond_Dénes: In ScyllaDB there are not such thing as too many partitions. I have never seen any production problem that resulted from the number of partitions.
When there is a problem with partitions, it is always related to their size. Both too large and too small (just a few bytes) can be a problem.
Ok, having very few partitions can cause problems, because it is impossible to spread the load across the cluster. But on the other end of the scale, I have not seen problems yet.
@hamonica: @Botond_Dénes Thank you very much, it is very helpful. I will think about how to further subdivide the partitions without causing problems in querying.