Is scylla bootstrapping inefficient?

horschi · October 9, 2023, 8:07am

Hi,

bootstrapping a new node causes a lot of compactions on the new node. Is this really necessary?

Couldn’t streaming be in token range order, where all ranges are written into one sstable? Also reshaping could be done directly when receiving, splitting data into shards.

From what I see, this is not how bootstrapping works, right?

Christian

avikivity · October 9, 2023, 9:25am

First, yes it’s inefficient.

Streaming is done in token range order. The problem is that each node contributes some vnodes, and those vnodes are discontiguous. Therefore there is a reshaping pass after all sstables are received, to reorganize the data into one sstable.

There should be one RESHAPE compaction per shard after bootstrap, no more.

With our upcoming tablets feature, there will be no reshaping or compaction needed.

horschi · October 9, 2023, 11:22am

But currently there is much more than one reshape/compaction?!

Couldn’t the bootstrapping node request token ranges in order and write those into the same sstables (one per shard)?
Right now it seems to be creating many small files, I assume because its writing a separate sstable per vnode.

Does this mean there will be no imrpovement for the vnode setup? There are many users out there using vnodes, I think it would be good if vnode bootstrapping still works well. Or will there be a migration path to tablets?

avikivity · October 9, 2023, 11:26am

That’s what there should be. If there is more, maybe there’s a bug, or I don’t understand the process will enough. @raphaelsc should know more.

We don’t request data in token order since that means only one NORMAL node will participate at a time. As it is, all NORMAL nodes (in the same rack) participate, and so they need to contribute less bandwidth.

For tablets, we plan to have an automatic migration process, though that will come some time after tablets are generally available.

horschi · October 9, 2023, 1:06pm

The new server could ask for multiple token ranges from different nodes and merge those into one sstable, resulting into something like this:
<Token 1 from server3><Token 5 from server8><Token 9 from server2><Token 11 from server4>…

Problems are error handling and concurrency. I assume thats why scylla is not doing it like this.

Avi, I apologize for distracting you so often! But bootstrapping really feels less optimal than in Cassandra and I’d like to understand it.

avikivity · October 9, 2023, 2:52pm

It can’t ask for the ranges serially, since that’s too slow. So it asks for the range in parallel, then merges them.

Maybe there’s a bug, but we need more detailed information about compaction history to see it.

horschi · October 9, 2023, 3:47pm

Just had a look at the logs from the weekend (grepped for one table for better readability), where I added a second node in that datacenter:

Oct 07 13:07:52 l-G-2 scylla[1957]:  [shard 0] schema_tables - Creating pc.tidx2 id=9a443dd0-a310-11e9-b8d6-000000000000 version=7b69fa40-eaba-3524-a0f0-d094b7adf0e2
Oct 07 13:25:08 l-G-2 scylla[1957]:  [shard 0] stream_session - [Stream #bb679ad0-6512-11ee-9f04-4c7028280052] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=44371279, received_partitions=38755775
Oct 07 13:25:42 l-G-2 scylla[1957]:  [shard 3] stream_session - [Stream #bb679ad0-6512-11ee-9f04-4c7028280052] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=44511404, received_partitions=39606479
Oct 07 13:26:01 l-G-2 scylla[1957]:  [shard 2] stream_session - [Stream #bb679ad0-6512-11ee-9f04-4c7028280052] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=46416445, received_partitions=39549278
Oct 07 13:27:18 l-G-2 scylla[1957]:  [shard 1] stream_session - [Stream #bb679ad0-6512-11ee-9f04-4c7028280052] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=54524645, received_partitions=39417776
Oct 07 13:43:10 l-G-2 scylla[1957]:  [shard 2] stream_session - [Stream #78daae20-6515-11ee-9f04-4c7028280052] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=43142986, received_partitions=34577049
Oct 07 13:43:24 l-G-2 scylla[1957]:  [shard 1] stream_session - [Stream #78daae20-6515-11ee-9f04-4c7028280052] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=42402039, received_partitions=34953502
Oct 07 13:43:33 l-G-2 scylla[1957]:  [shard 3] stream_session - [Stream #78daae20-6515-11ee-9f04-4c7028280052] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=42594868, received_partitions=35263037
Oct 07 13:43:54 l-G-2 scylla[1957]:  [shard 0] stream_session - [Stream #78daae20-6515-11ee-9f04-4c7028280052] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=44591343, received_partitions=35058960
Oct 07 13:58:15 l-G-2 scylla[1957]:  [shard 2] stream_session - [Stream #cab4e510-6517-11ee-9f04-4c7028280052] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=32536122, received_partitions=28806530
Oct 07 13:58:29 l-G-2 scylla[1957]:  [shard 1] stream_session - [Stream #cab4e510-6517-11ee-9f04-4c7028280052] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=32564732, received_partitions=29040595
Oct 07 13:58:35 l-G-2 scylla[1957]:  [shard 0] stream_session - [Stream #cab4e510-6517-11ee-9f04-4c7028280052] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=31981548, received_partitions=27956277
Oct 07 13:58:45 l-G-2 scylla[1957]:  [shard 3] stream_session - [Stream #cab4e510-6517-11ee-9f04-4c7028280052] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=31517774, received_partitions=28723376
Oct 07 14:16:11 l-G-2 scylla[1957]:  [shard 3] stream_session - [Stream #e3f228b0-6519-11ee-9f04-4c7028280052] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=39150632, received_partitions=34217934
Oct 07 14:16:13 l-G-2 scylla[1957]:  [shard 1] stream_session - [Stream #e3f228b0-6519-11ee-9f04-4c7028280052] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=41029890, received_partitions=35476968
Oct 07 14:16:14 l-G-2 scylla[1957]:  [shard 2] stream_session - [Stream #e3f228b0-6519-11ee-9f04-4c7028280052] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=40157781, received_partitions=34926570
Oct 07 14:16:18 l-G-2 scylla[1957]:  [shard 0] stream_session - [Stream #e3f228b0-6519-11ee-9f04-4c7028280052] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=41193876, received_partitions=34876102
Oct 07 14:32:29 l-G-2 scylla[1957]:  [shard 0] stream_session - [Stream #5b5a0560-651c-11ee-9f04-4c7028280052] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=37938009, received_partitions=33933323
Oct 07 14:33:12 l-G-2 scylla[1957]:  [shard 1] stream_session - [Stream #5b5a0560-651c-11ee-9f04-4c7028280052] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=38031656, received_partitions=33993599
Oct 07 14:33:14 l-G-2 scylla[1957]:  [shard 3] stream_session - [Stream #5b5a0560-651c-11ee-9f04-4c7028280052] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=39592744, received_partitions=35009083
Oct 07 14:33:15 l-G-2 scylla[1957]:  [shard 2] stream_session - [Stream #5b5a0560-651c-11ee-9f04-4c7028280052] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=38748607, received_partitions=34468116
Oct 07 14:50:18 l-G-2 scylla[1957]:  [shard 2] stream_session - [Stream #b3acbad0-651e-11ee-9f04-4c7028280052] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=40288700, received_partitions=36808154
Oct 07 14:50:46 l-G-2 scylla[1957]:  [shard 1] stream_session - [Stream #b3acbad0-651e-11ee-9f04-4c7028280052] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=40921581, received_partitions=36969910
Oct 07 14:51:00 l-G-2 scylla[1957]:  [shard 0] stream_session - [Stream #b3acbad0-651e-11ee-9f04-4c7028280052] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=41321649, received_partitions=36571644
Oct 07 14:51:07 l-G-2 scylla[1957]:  [shard 3] stream_session - [Stream #b3acbad0-651e-11ee-9f04-4c7028280052] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=41578576, received_partitions=37378731
Oct 07 15:04:27 l-G-2 scylla[1957]:  [shard 1] stream_session - [Stream #3a97d820-6521-11ee-9f04-4c7028280052] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=30342879, received_partitions=25436020
Oct 07 15:04:41 l-G-2 scylla[1957]:  [shard 0] stream_session - [Stream #3a97d820-6521-11ee-9f04-4c7028280052] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=31429115, received_partitions=26228088
Oct 07 15:04:48 l-G-2 scylla[1957]:  [shard 2] stream_session - [Stream #3a97d820-6521-11ee-9f04-4c7028280052] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=30913161, received_partitions=25776572
Oct 07 15:04:58 l-G-2 scylla[1957]:  [shard 3] stream_session - [Stream #3a97d820-6521-11ee-9f04-4c7028280052] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=32498112, received_partitions=26977850
Oct 07 15:22:38 l-G-2 scylla[1957]:  [shard 0] stream_session - [Stream #187eadc0-6523-11ee-9f04-4c7028280052] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=40400018, received_partitions=37994129
Oct 07 15:22:41 l-G-2 scylla[1957]:  [shard 1] stream_session - [Stream #187eadc0-6523-11ee-9f04-4c7028280052] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=40911080, received_partitions=38092502
Oct 07 15:22:58 l-G-2 scylla[1957]:  [shard 3] stream_session - [Stream #187eadc0-6523-11ee-9f04-4c7028280052] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=41529185, received_partitions=39324692
Oct 07 15:23:05 l-G-2 scylla[1957]:  [shard 2] stream_session - [Stream #187eadc0-6523-11ee-9f04-4c7028280052] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=43119767, received_partitions=39077318
Oct 07 15:38:38 l-G-2 scylla[1957]:  [shard 1] stream_session - [Stream #b14c0a50-6525-11ee-9f04-4c7028280052] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=36230105, received_partitions=30900074
Oct 07 15:38:51 l-G-2 scylla[1957]:  [shard 0] stream_session - [Stream #b14c0a50-6525-11ee-9f04-4c7028280052] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=36218474, received_partitions=30155405
Oct 07 15:38:57 l-G-2 scylla[1957]:  [shard 2] stream_session - [Stream #b14c0a50-6525-11ee-9f04-4c7028280052] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=37106328, received_partitions=31043703
Oct 07 15:39:07 l-G-2 scylla[1957]:  [shard 3] stream_session - [Stream #b14c0a50-6525-11ee-9f04-4c7028280052] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=36193007, received_partitions=30840935
Oct 07 15:57:18 l-G-2 scylla[1957]:  [shard 1] stream_session - [Stream #e8a82e50-6527-11ee-9f04-4c7028280052] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=44050522, received_partitions=36538552
Oct 07 15:57:40 l-G-2 scylla[1957]:  [shard 2] stream_session - [Stream #e8a82e50-6527-11ee-9f04-4c7028280052] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=44104559, received_partitions=36747050
Oct 07 15:57:53 l-G-2 scylla[1957]:  [shard 0] stream_session - [Stream #e8a82e50-6527-11ee-9f04-4c7028280052] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=44493080, received_partitions=36095232
Oct 07 15:58:03 l-G-2 scylla[1957]:  [shard 3] stream_session - [Stream #e8a82e50-6527-11ee-9f04-4c7028280052] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=45978673, received_partitions=37643025
Oct 07 16:00:46 l-G-2 scylla[1957]:  [shard 1] stream_session - [Stream #985d7ec0-652a-11ee-9f04-4c7028280052] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=1773971, received_partitions=1559143
Oct 07 16:00:54 l-G-2 scylla[1957]:  [shard 2] stream_session - [Stream #985d7ec0-652a-11ee-9f04-4c7028280052] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=1876830, received_partitions=1511903
Oct 07 16:00:59 l-G-2 scylla[1957]:  [shard 0] stream_session - [Stream #985d7ec0-652a-11ee-9f04-4c7028280052] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=2129439, received_partitions=1675531
Oct 07 16:01:02 l-G-2 scylla[1957]:  [shard 3] stream_session - [Stream #985d7ec0-652a-11ee-9f04-4c7028280052] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=1836312, received_partitions=1512729

Oct 07 16:01:08 l-G-2 scylla[1957]:  [shard 0] storage_service - Bootstrap completed!

Oct 07 16:05:47 l-G-2 scylla[1957]:  [shard 1] table - Starting off-strategy compaction for pc.tidx2, 11 candidates were found
Oct 07 16:05:47 l-G-2 scylla[1957]:  [shard 1] compaction - [Reshape pc.tidx2 606820a0-652b-11ee-8637-4c7128280052] Reshaping [/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-5-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-17-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-41-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-13-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-29-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-25-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-37-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-33-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-9-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-21-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-45-big-Data.db:level=0:origin=streaming]
Oct 07 16:05:53 l-G-2 scylla[1957]:  [shard 2] table - Starting off-strategy compaction for pc.tidx2, 11 candidates were found
Oct 07 16:05:53 l-G-2 scylla[1957]:  [shard 2] compaction - [Reshape pc.tidx2 6488cc70-652b-11ee-a2f2-4c7228280052] Reshaping [/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-6-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-18-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-42-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-14-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-30-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-26-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-38-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-34-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-10-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-22-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-46-big-Data.db:level=0:origin=streaming]
Oct 07 16:05:56 l-G-2 scylla[1957]:  [shard 3] table - Starting off-strategy compaction for pc.tidx2, 11 candidates were found
Oct 07 16:05:56 l-G-2 scylla[1957]:  [shard 3] compaction - [Reshape pc.tidx2 66013ba0-652b-11ee-94ee-4c7328280052] Reshaping [/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-7-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-19-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-43-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-15-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-31-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-27-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-39-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-35-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-11-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-23-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-47-big-Data.db:level=0:origin=streaming]
Oct 07 16:06:03 l-G-2 scylla[1957]:  [shard 0] table - Starting off-strategy compaction for pc.tidx2, 11 candidates were found
Oct 07 16:06:03 l-G-2 scylla[1957]:  [shard 0] compaction - [Reshape pc.tidx2 6a7efb90-652b-11ee-9f04-4c7028280052] Reshaping [/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-4-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-16-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-40-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-12-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-28-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-24-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-36-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-32-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-8-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-20-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-44-big-Data.db:level=0:origin=streaming]
Oct 07 17:03:14 l-G-2 scylla[1957]:  [shard 0] compaction - [Reshape pc.tidx2 6a7efb90-652b-11ee-9f04-4c7028280052] Reshaped 11 sstables to [/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-48-big-Data.db:level=0]. 35GB to 35GB (~100% of original) in 3430348ms = 10MB/s. ~339312640 total partitions merged to 339300466.
Oct 07 17:03:14 l-G-2 scylla[1957]:  [shard 0] table - Done with off-strategy compaction for pc.tidx2
Oct 07 17:03:59 l-G-2 scylla[1957]:  [shard 2] compaction - [Reshape pc.tidx2 6488cc70-652b-11ee-a2f2-4c7228280052] Reshaped 11 sstables to [/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-50-big-Data.db:level=0]. 36GB to 36GB (~100% of original) in 3485695ms = 10MB/s. ~343304192 total partitions merged to 343292243.
Oct 07 17:03:59 l-G-2 scylla[1957]:  [shard 2] table - Done with off-strategy compaction for pc.tidx2
Oct 07 17:04:44 l-G-2 scylla[1957]:  [shard 3] compaction - [Reshape pc.tidx2 66013ba0-652b-11ee-94ee-4c7328280052] Reshaped 11 sstables to [/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-51-big-Data.db:level=0]. 36GB to 36GB (~100% of original) in 3527867ms = 10MB/s. ~346511360 total partitions merged to 346497871.
Oct 07 17:04:44 l-G-2 scylla[1957]:  [shard 3] table - Done with off-strategy compaction for pc.tidx2
Oct 07 17:05:35 l-G-2 scylla[1957]:  [shard 1] compaction - [Reshape pc.tidx2 606820a0-652b-11ee-8637-4c7128280052] Reshaped 11 sstables to [/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-49-big-Data.db:level=0]. 36GB to 36GB (~100% of original) in 3588823ms = 10MB/s. ~342386688 total partitions merged to 342378641.
Oct 07 17:05:35 l-G-2 scylla[1957]:  [shard 1] table - Done with off-strategy compaction for pc.tidx2
Oct 07 17:27:58 l-G-2 scylla[1957]:  [shard 0] table - Starting off-strategy compaction for pc.tidx2, 18 candidates were found
Oct 07 17:27:58 l-G-2 scylla[1957]:  [shard 0] compaction - [Reshape pc.tidx2 db921f52-6536-11ee-9f04-4c7028280052] Reshaping [/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-52-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-56-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-60-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-64-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-68-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-72-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-76-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-80-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-84-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-88-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-92-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-96-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-100-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-104-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-108-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-112-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-116-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-120-big-Data.db:level=0:origin=repair]
Oct 07 17:28:07 l-G-2 scylla[1957]:  [shard 0] compaction - [Reshape pc.tidx2 db921f52-6536-11ee-9f04-4c7028280052] Reshaped 18 sstables to [/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-124-big-Data.db:level=0]. 72MB to 73MB (~100% of original) in 9557ms = 7MB/s. ~751616 total partitions merged to 735389.
Oct 07 17:28:07 l-G-2 scylla[1957]:  [shard 0] table - Done with off-strategy compaction for pc.tidx2
Oct 07 17:28:07 l-G-2 scylla[1957]:  [shard 0] table - Starting off-strategy compaction for pc.tidx2, 0 candidates were found
Oct 07 17:28:07 l-G-2 scylla[1957]:  [shard 0] table - Done with off-strategy compaction for pc.tidx2
Oct 07 17:28:07 l-G-2 scylla[1957]:  [shard 0] table - Starting off-strategy compaction for pc.tidx2, 0 candidates were found
Oct 07 17:28:07 l-G-2 scylla[1957]:  [shard 0] table - Done with off-strategy compaction for pc.tidx2
Oct 07 17:28:07 l-G-2 scylla[1957]:  [shard 0] table - Starting off-strategy compaction for pc.tidx2, 0 candidates were found
Oct 07 17:28:07 l-G-2 scylla[1957]:  [shard 0] table - Done with off-strategy compaction for pc.tidx2
Oct 07 17:28:07 l-G-2 scylla[1957]:  [shard 0] table - Starting off-strategy compaction for pc.tidx2, 0 candidates were found
Oct 07 17:28:07 l-G-2 scylla[1957]:  [shard 0] table - Done with off-strategy compaction for pc.tidx2
Oct 07 17:28:53 l-G-2 scylla[1957]:  [shard 2] table - Starting off-strategy compaction for pc.tidx2, 16 candidates were found
Oct 07 17:28:53 l-G-2 scylla[1957]:  [shard 2] compaction - [Reshape pc.tidx2 fcbabd44-6536-11ee-a2f2-4c7228280052] Reshaping [/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-54-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-58-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-62-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-66-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-70-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-74-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-78-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-82-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-86-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-90-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-94-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-98-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-102-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-106-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-110-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-114-big-Data.db:level=0:origin=repair]
Oct 07 17:28:58 l-G-2 scylla[1957]:  [shard 2] compaction - [Reshape pc.tidx2 fcbabd44-6536-11ee-a2f2-4c7228280052] Reshaped 16 sstables to [/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-118-big-Data.db:level=0]. 54MB to 54MB (~99% of original) in 5163ms = 10MB/s. ~442368 total partitions merged to 425379.
Oct 07 17:28:58 l-G-2 scylla[1957]:  [shard 2] table - Done with off-strategy compaction for pc.tidx2
Oct 07 17:28:58 l-G-2 scylla[1957]:  [shard 2] table - Starting off-strategy compaction for pc.tidx2, 0 candidates were found
Oct 07 17:28:58 l-G-2 scylla[1957]:  [shard 2] table - Done with off-strategy compaction for pc.tidx2
Oct 07 17:28:58 l-G-2 scylla[1957]:  [shard 2] table - Starting off-strategy compaction for pc.tidx2, 0 candidates were found
Oct 07 17:28:58 l-G-2 scylla[1957]:  [shard 2] table - Done with off-strategy compaction for pc.tidx2
Oct 07 17:28:58 l-G-2 scylla[1957]:  [shard 2] table - Starting off-strategy compaction for pc.tidx2, 0 candidates were found
Oct 07 17:28:58 l-G-2 scylla[1957]:  [shard 2] table - Done with off-strategy compaction for pc.tidx2
Oct 07 17:28:58 l-G-2 scylla[1957]:  [shard 2] table - Starting off-strategy compaction for pc.tidx2, 0 candidates were found
Oct 07 17:28:58 l-G-2 scylla[1957]:  [shard 2] table - Done with off-strategy compaction for pc.tidx2
Oct 07 17:29:10 l-G-2 scylla[1957]:  [shard 3] table - Starting off-strategy compaction for pc.tidx2, 17 candidates were found
Oct 07 17:29:10 l-G-2 scylla[1957]:  [shard 3] compaction - [Reshape pc.tidx2 06c95ad3-6537-11ee-94ee-4c7328280052] Reshaping [/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-55-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-59-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-63-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-67-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-71-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-75-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-79-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-83-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-87-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-91-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-95-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-99-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-103-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-107-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-111-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-115-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-119-big-Data.db:level=0:origin=repair]
Oct 07 17:29:15 l-G-2 scylla[1957]:  [shard 3] compaction - [Reshape pc.tidx2 06c95ad3-6537-11ee-94ee-4c7328280052] Reshaped 17 sstables to [/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-123-big-Data.db:level=0]. 46MB to 46MB (~100% of original) in 4759ms = 9MB/s. ~448512 total partitions merged to 432120.
Oct 07 17:29:15 l-G-2 scylla[1957]:  [shard 3] table - Done with off-strategy compaction for pc.tidx2
Oct 07 17:29:15 l-G-2 scylla[1957]:  [shard 3] table - Starting off-strategy compaction for pc.tidx2, 0 candidates were found
Oct 07 17:29:15 l-G-2 scylla[1957]:  [shard 3] table - Done with off-strategy compaction for pc.tidx2
Oct 07 17:29:15 l-G-2 scylla[1957]:  [shard 3] table - Starting off-strategy compaction for pc.tidx2, 0 candidates were found
Oct 07 17:29:15 l-G-2 scylla[1957]:  [shard 3] table - Done with off-strategy compaction for pc.tidx2
Oct 07 17:29:15 l-G-2 scylla[1957]:  [shard 3] table - Starting off-strategy compaction for pc.tidx2, 0 candidates were found
Oct 07 17:29:15 l-G-2 scylla[1957]:  [shard 3] table - Done with off-strategy compaction for pc.tidx2
Oct 07 17:29:15 l-G-2 scylla[1957]:  [shard 3] table - Starting off-strategy compaction for pc.tidx2, 0 candidates were found
Oct 07 17:29:15 l-G-2 scylla[1957]:  [shard 3] table - Done with off-strategy compaction for pc.tidx2
Oct 07 17:29:34 l-G-2 scylla[1957]:  [shard 1] table - Starting off-strategy compaction for pc.tidx2, 20 candidates were found
Oct 07 17:29:34 l-G-2 scylla[1957]:  [shard 1] compaction - [Reshape pc.tidx2 15416d02-6537-11ee-8637-4c7128280052] Reshaping [/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-53-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-57-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-61-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-65-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-69-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-73-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-77-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-81-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-85-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-89-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-93-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-97-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-101-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-105-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-109-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-113-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-117-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-121-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-125-big-Data.db:level=0:origin=repair,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-129-big-Data.db:level=0:origin=repair]
Oct 07 17:29:37 l-G-2 scylla[1957]:  [shard 1] compaction - [Reshape pc.tidx2 15416d02-6537-11ee-8637-4c7128280052] Reshaped 20 sstables to [/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-133-big-Data.db:level=0]. 37MB to 37MB (~99% of original) in 2861ms = 13MB/s. ~346112 total partitions merged to 327368.
Oct 07 17:29:37 l-G-2 scylla[1957]:  [shard 1] table - Done with off-strategy compaction for pc.tidx2
Oct 07 17:29:37 l-G-2 scylla[1957]:  [shard 1] table - Starting off-strategy compaction for pc.tidx2, 0 candidates were found
Oct 07 17:29:37 l-G-2 scylla[1957]:  [shard 1] table - Done with off-strategy compaction for pc.tidx2
Oct 07 17:29:37 l-G-2 scylla[1957]:  [shard 1] table - Starting off-strategy compaction for pc.tidx2, 0 candidates were found
Oct 07 17:29:37 l-G-2 scylla[1957]:  [shard 1] table - Done with off-strategy compaction for pc.tidx2
Oct 07 17:29:37 l-G-2 scylla[1957]:  [shard 1] table - Starting off-strategy compaction for pc.tidx2, 0 candidates were found
Oct 07 17:29:37 l-G-2 scylla[1957]:  [shard 1] table - Done with off-strategy compaction for pc.tidx2
Oct 07 17:29:37 l-G-2 scylla[1957]:  [shard 1] table - Starting off-strategy compaction for pc.tidx2, 0 candidates were found
Oct 07 17:29:37 l-G-2 scylla[1957]:  [shard 1] table - Done with off-strategy compaction for pc.tidx2
Oct 07 17:29:37 l-G-2 scylla[1957]:  [shard 1] table - Starting off-strategy compaction for pc.tidx2, 0 candidates were found
Oct 07 17:29:37 l-G-2 scylla[1957]:  [shard 1] table - Done with off-strategy compaction for pc.tidx2
Oct 07 17:35:26 l-G-2 scylla[1957]:  [shard 3] table - Starting off-strategy compaction for pc.tidx2, 1 candidates were found
Oct 07 17:35:26 l-G-2 scylla[1957]:  [shard 3] table - Done with off-strategy compaction for pc.tidx2
Oct 07 17:35:27 l-G-2 scylla[1957]:  [shard 1] table - Starting off-strategy compaction for pc.tidx2, 1 candidates were found
Oct 07 17:35:27 l-G-2 scylla[1957]:  [shard 1] table - Done with off-strategy compaction for pc.tidx2
Oct 07 17:35:28 l-G-2 scylla[1957]:  [shard 0] table - Starting off-strategy compaction for pc.tidx2, 1 candidates were found
Oct 07 17:35:28 l-G-2 scylla[1957]:  [shard 0] table - Done with off-strategy compaction for pc.tidx2
Oct 07 17:35:30 l-G-2 scylla[1957]:  [shard 2] table - Starting off-strategy compaction for pc.tidx2, 1 candidates were found
Oct 07 17:35:30 l-G-2 scylla[1957]:  [shard 2] table - Done with off-strategy compaction for pc.tidx2

13:07:52 - 16:01:08 → Bootstrap/streaming (~ 3 hours)
16:05:47 - 17:35:30 → Reshape/compact (1 1/2 hours)

This was bootstrapping from one other node in the datacenter. I assume streaming would have been faster if there were multiple nodes in that datacenter. I have to see if I can find some logs for that.

Also this was on spinning disks.

horschi · October 9, 2023, 4:13pm

I found another log of a bootstrap into a larger cluster (adding a 5th node to a rack) and with SSDs:


Oct 07 11:59:54 op-L3-5 scylla[16180]:  [shard 0] schema_tables - Creating pc.tidx2 id=9a443dd0-a310-11e9-b8d6-000000000000 version=55574014-2efd-3cef-8645-307079518f31
Oct 07 12:03:27 op-L3-5 scylla[16180]:  [shard 1] stream_session - [Stream #3b4f2e20-6509-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=4488497, received_partitions=3741732
Oct 07 12:03:36 op-L3-5 scylla[16180]:  [shard 2] stream_session - [Stream #3b4f2e20-6509-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=4778921, received_partitions=4007925
Oct 07 12:03:47 op-L3-5 scylla[16180]:  [shard 0] stream_session - [Stream #3b4f2e20-6509-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=5346474, received_partitions=4738556
Oct 07 12:03:51 op-L3-5 scylla[16180]:  [shard 3] stream_session - [Stream #3b4f2e20-6509-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=4905606, received_partitions=4163392
Oct 07 12:03:53 op-L3-5 scylla[16180]:  [shard 4] stream_session - [Stream #3b4f2e20-6509-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=5500548, received_partitions=4837999
Oct 07 12:03:57 op-L3-5 scylla[16180]:  [shard 5] stream_session - [Stream #3b4f2e20-6509-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=5966219, received_partitions=5074706
Oct 07 12:04:22 op-L3-5 scylla[16180]:  [shard 3] stream_session - [Stream #3b4f2e22-6509-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=5195854, received_partitions=5282879
Oct 07 12:04:22 op-L3-5 scylla[16180]:  [shard 4] stream_session - [Stream #3b4f2e22-6509-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=5278291, received_partitions=5438277
Oct 07 12:04:26 op-L3-5 scylla[16180]:  [shard 2] stream_session - [Stream #3b4f2e22-6509-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=5785530, received_partitions=5700299
Oct 07 12:04:26 op-L3-5 scylla[16180]:  [shard 5] stream_session - [Stream #3b4f2e22-6509-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=5639641, received_partitions=5755052
Oct 07 12:04:28 op-L3-5 scylla[16180]:  [shard 0] stream_session - [Stream #3b4f2e22-6509-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=5579620, received_partitions=5751991
Oct 07 12:04:28 op-L3-5 scylla[16180]:  [shard 1] stream_session - [Stream #3b4f2e22-6509-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=5710304, received_partitions=5724911
Oct 07 12:05:08 op-L3-5 scylla[16180]:  [shard 2] stream_session - [Stream #3b4f2e21-6509-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=7667630, received_partitions=6705163
Oct 07 12:05:08 op-L3-5 scylla[16180]:  [shard 0] stream_session - [Stream #3b4f2e21-6509-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=8292818, received_partitions=7153566
Oct 07 12:05:08 op-L3-5 scylla[16180]:  [shard 1] stream_session - [Stream #3b4f2e21-6509-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=8068853, received_partitions=6706743
Oct 07 12:05:14 op-L3-5 scylla[16180]:  [shard 3] stream_session - [Stream #3b4f2e21-6509-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=7932116, received_partitions=6679997
Oct 07 12:05:17 op-L3-5 scylla[16180]:  [shard 4] stream_session - [Stream #3b4f2e21-6509-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=9189998, received_partitions=7468994
Oct 07 12:05:25 op-L3-5 scylla[16180]:  [shard 0] stream_session - [Stream #3b4f2e23-6509-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=7042159, received_partitions=6061478
Oct 07 12:05:29 op-L3-5 scylla[16180]:  [shard 1] stream_session - [Stream #3b4f2e23-6509-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=7493739, received_partitions=6334634
Oct 07 12:05:32 op-L3-5 scylla[16180]:  [shard 5] stream_session - [Stream #3b4f2e21-6509-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=9746087, received_partitions=8088443
Oct 07 12:05:35 op-L3-5 scylla[16180]:  [shard 3] stream_session - [Stream #3b4f2e23-6509-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=7855459, received_partitions=6510260
Oct 07 12:05:35 op-L3-5 scylla[16180]:  [shard 4] stream_session - [Stream #3b4f2e23-6509-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=7703930, received_partitions=6565536
Oct 07 12:05:35 op-L3-5 scylla[16180]:  [shard 2] stream_session - [Stream #3b4f2e23-6509-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=7828306, received_partitions=6425801
Oct 07 12:05:45 op-L3-5 scylla[16180]:  [shard 5] stream_session - [Stream #3b4f2e23-6509-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=7762524, received_partitions=6468571
Oct 07 12:07:45 op-L3-5 scylla[16180]:  [shard 2] stream_session - [Stream #ab9bce90-6509-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=6942207, received_partitions=5413327
Oct 07 12:07:46 op-L3-5 scylla[16180]:  [shard 3] stream_session - [Stream #ab9bce90-6509-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=6436323, received_partitions=5351286
Oct 07 12:07:46 op-L3-5 scylla[16180]:  [shard 1] stream_session - [Stream #ab9bce90-6509-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=6650647, received_partitions=5500811
Oct 07 12:07:48 op-L3-5 scylla[16180]:  [shard 0] stream_session - [Stream #ab9bce90-6509-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=7011265, received_partitions=6173794
Oct 07 12:07:57 op-L3-5 scylla[16180]:  [shard 2] stream_session - [Stream #c017a470-6509-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=5213995, received_partitions=5332720
Oct 07 12:07:59 op-L3-5 scylla[16180]:  [shard 5] stream_session - [Stream #ab9bce90-6509-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=7967082, received_partitions=6752218
Oct 07 12:08:00 op-L3-5 scylla[16180]:  [shard 4] stream_session - [Stream #ab9bce90-6509-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=7988809, received_partitions=6698485
Oct 07 12:08:04 op-L3-5 scylla[16180]:  [shard 3] stream_session - [Stream #c017a470-6509-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=5313083, received_partitions=5209462
Oct 07 12:08:05 op-L3-5 scylla[16180]:  [shard 4] stream_session - [Stream #c017a470-6509-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=5504293, received_partitions=5461453
Oct 07 12:08:15 op-L3-5 scylla[16180]:  [shard 1] stream_session - [Stream #c017a470-6509-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=6360454, received_partitions=6321548
Oct 07 12:08:16 op-L3-5 scylla[16180]:  [shard 5] stream_session - [Stream #c017a470-6509-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=5817204, received_partitions=5918249
Oct 07 12:08:20 op-L3-5 scylla[16180]:  [shard 0] stream_session - [Stream #c017a470-6509-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=6595192, received_partitions=6195387
Oct 07 12:09:18 op-L3-5 scylla[16180]:  [shard 2] stream_session - [Stream #ed0ac0c0-6509-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=6768063, received_partitions=5976709
Oct 07 12:09:23 op-L3-5 scylla[16180]:  [shard 1] stream_session - [Stream #ed0ac0c0-6509-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=7162522, received_partitions=5699596
Oct 07 12:09:23 op-L3-5 scylla[16180]:  [shard 0] stream_session - [Stream #ed0ac0c0-6509-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=6728702, received_partitions=5457614
Oct 07 12:09:28 op-L3-5 scylla[16180]:  [shard 3] stream_session - [Stream #ed0ac0c0-6509-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=7389654, received_partitions=5746637
Oct 07 12:09:30 op-L3-5 scylla[16180]:  [shard 4] stream_session - [Stream #ed0ac0c0-6509-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=7922550, received_partitions=6008651
Oct 07 12:09:31 op-L3-5 scylla[16180]:  [shard 5] stream_session - [Stream #ed0ac0c0-6509-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=7437238, received_partitions=5774987
Oct 07 12:10:15 op-L3-5 scylla[16180]:  [shard 0] stream_session - [Stream #40adbbb0-650a-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=3183761, received_partitions=2810414
Oct 07 12:10:31 op-L3-5 scylla[16180]:  [shard 5] stream_session - [Stream #40adbbb0-650a-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=4111675, received_partitions=3194534
Oct 07 12:10:39 op-L3-5 scylla[16180]:  [shard 1] stream_session - [Stream #40adbbb0-650a-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=4317173, received_partitions=3824106
Oct 07 12:10:41 op-L3-5 scylla[16180]:  [shard 2] stream_session - [Stream #40adbbb0-650a-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=5126740, received_partitions=4554028
Oct 07 12:10:42 op-L3-5 scylla[16180]:  [shard 4] stream_session - [Stream #40adbbb0-650a-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=4776617, received_partitions=4016289
Oct 07 12:10:48 op-L3-5 scylla[16180]:  [shard 3] stream_session - [Stream #40adbbb0-650a-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=5063343, received_partitions=4614664
Oct 07 12:11:04 op-L3-5 scylla[16180]:  [shard 0] stream_session - [Stream #f1d6c7c0-6509-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=8216208, received_partitions=7356680
Oct 07 12:11:06 op-L3-5 scylla[16180]:  [shard 3] stream_session - [Stream #f1d6c7c0-6509-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=8537428, received_partitions=7567256
Oct 07 12:11:15 op-L3-5 scylla[16180]:  [shard 4] stream_session - [Stream #f1d6c7c0-6509-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=8419992, received_partitions=7645381
Oct 07 12:11:21 op-L3-5 scylla[16180]:  [shard 1] stream_session - [Stream #f1d6c7c0-6509-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=8904799, received_partitions=7968961
Oct 07 12:11:29 op-L3-5 scylla[16180]:  [shard 2] stream_session - [Stream #f1d6c7c0-6509-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=9604728, received_partitions=8327918
Oct 07 12:11:29 op-L3-5 scylla[16180]:  [shard 5] stream_session - [Stream #f1d6c7c0-6509-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=8826331, received_partitions=7544011
Oct 07 12:11:59 op-L3-5 scylla[16180]:  [shard 0] stream_session - [Stream #4af43220-650a-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=5772298, received_partitions=5859133
Oct 07 12:12:17 op-L3-5 scylla[16180]:  [shard 1] stream_session - [Stream #4af43220-650a-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=6179111, received_partitions=6478107
Oct 07 12:12:19 op-L3-5 scylla[16180]:  [shard 5] stream_session - [Stream #4af43220-650a-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=6161734, received_partitions=6168741
...
Oct 07 12:19:51 op-L3-5 scylla[16180]:  [shard 5] stream_session - [Stream #76ddb130-650b-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=4781148, received_partitions=3813919
Oct 07 12:20:01 op-L3-5 scylla[16180]:  [shard 2] stream_session - [Stream #76ddb130-650b-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=5519006, received_partitions=4436153
Oct 07 12:20:01 op-L3-5 scylla[16180]:  [shard 3] stream_session - [Stream #76ddb130-650b-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=5451976, received_partitions=4317423
Oct 07 12:20:05 op-L3-5 scylla[16180]:  [shard 1] stream_session - [Stream #76ddb130-650b-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=5624326, received_partitions=4874916
Oct 07 12:22:55 op-L3-5 scylla[16180]:  [shard 2] stream_session - [Stream #993a8960-650b-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=9482753, received_partitions=7603396
Oct 07 12:22:56 op-L3-5 scylla[16180]:  [shard 3] stream_session - [Stream #de39f550-650b-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=5599635, received_partitions=4757963
Oct 07 12:22:57 op-L3-5 scylla[16180]:  [shard 2] stream_session - [Stream #de39f550-650b-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=6443785, received_partitions=5103227
Oct 07 12:23:03 op-L3-5 scylla[16180]:  [shard 1] stream_session - [Stream #993a8960-650b-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=10798934, received_partitions=7702842
Oct 07 12:23:07 op-L3-5 scylla[16180]:  [shard 1] stream_session - [Stream #de39f550-650b-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=6333377, received_partitions=5227758
Oct 07 12:23:07 op-L3-5 scylla[16180]:  [shard 3] stream_session - [Stream #993a8960-650b-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=10721938, received_partitions=7489243
Oct 07 12:23:09 op-L3-5 scylla[16180]:  [shard 5] stream_session - [Stream #993a8960-650b-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=11372183, received_partitions=8383114
Oct 07 12:23:09 op-L3-5 scylla[16180]:  [shard 4] stream_session - [Stream #de39f550-650b-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=6971240, received_partitions=5667395
Oct 07 12:23:10 op-L3-5 scylla[16180]:  [shard 5] stream_session - [Stream #de39f550-650b-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=7236204, received_partitions=5979031
Oct 07 12:23:15 op-L3-5 scylla[16180]:  [shard 0] stream_session - [Stream #993a8960-650b-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=11865671, received_partitions=8293860
Oct 07 12:23:16 op-L3-5 scylla[16180]:  [shard 4] stream_session - [Stream #993a8960-650b-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=12461045, received_partitions=8165765
Oct 07 12:23:16 op-L3-5 scylla[16180]:  [shard 0] stream_session - [Stream #de39f550-650b-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=7256489, received_partitions=6312671
Oct 07 12:23:44 op-L3-5 scylla[16180]:  [shard 3] stream_session - [Stream #cfc8e800-650b-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=6864416, received_partitions=7336250
Oct 07 12:23:49 op-L3-5 scylla[16180]:  [shard 1] stream_session - [Stream #cfc8e800-650b-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=7073736, received_partitions=7236051
Oct 07 12:23:54 op-L3-5 scylla[16180]:  [shard 2] stream_session - [Stream #cfc8e800-650b-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=6975372, received_partitions=7217043
Oct 07 12:23:57 op-L3-5 scylla[16180]:  [shard 4] stream_session - [Stream #cfc8e800-650b-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=7427317, received_partitions=7717180
Oct 07 12:23:59 op-L3-5 scylla[16180]:  [shard 5] stream_session - [Stream #cfc8e800-650b-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=7685601, received_partitions=7984992
Oct 07 12:23:59 op-L3-5 scylla[16180]:  [shard 0] stream_session - [Stream #cfc8e800-650b-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=7832874, received_partitions=7989933
Oct 07 12:25:17 op-L3-5 scylla[16180]:  [shard 4] stream_session - [Stream #ee46b140-650b-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=7980226, received_partitions=6517564
Oct 07 12:25:30 op-L3-5 scylla[16180]:  [shard 0] stream_session - [Stream #ee46b140-650b-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=8040659, received_partitions=6494326
Oct 07 12:25:31 op-L3-5 scylla[16180]:  [shard 1] stream_session - [Stream #ee46b140-650b-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=8935070, received_partitions=7382025
Oct 07 12:25:31 op-L3-5 scylla[16180]:  [shard 5] stream_session - [Stream #ee46b140-650b-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=7969760, received_partitions=6515825
Oct 07 12:25:36 op-L3-5 scylla[16180]:  [shard 3] stream_session - [Stream #ee46b140-650b-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=9046637, received_partitions=7370881
Oct 07 12:25:36 op-L3-5 scylla[16180]:  [shard 2] stream_session - [Stream #ee46b140-650b-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=9520764, received_partitions=7816445
Oct 07 12:27:18 op-L3-5 scylla[16180]:  [shard 4] stream_session - [Stream #83861700-650c-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=4681855, received_partitions=4914885
Oct 07 12:27:26 op-L3-5 scylla[16180]:  [shard 0] stream_session - [Stream #83861700-650c-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=4746021, received_partitions=4761231
Oct 07 12:27:27 op-L3-5 scylla[16180]:  [shard 5] stream_session - [Stream #83861700-650c-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=4865779, received_partitions=5062616
Oct 07 12:27:29 op-L3-5 scylla[16180]:  [shard 3] stream_session - [Stream #83861700-650c-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=4535462, received_partitions=4706415
Oct 07 12:27:31 op-L3-5 scylla[16180]:  [shard 2] stream_session - [Stream #83861700-650c-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=4643749, received_partitions=4678749
Oct 07 12:27:34 op-L3-5 scylla[16180]:  [shard 1] stream_session - [Stream #83861700-650c-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=4979556, received_partitions=4920737
Oct 07 12:27:41 op-L3-5 scylla[16180]:  [shard 2] stream_session - [Stream #61b3f2a0-650c-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=7909066, received_partitions=6654180
Oct 07 12:27:54 op-L3-5 scylla[16180]:  [shard 3] stream_session - [Stream #61b3f2a0-650c-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=7962019, received_partitions=7072091
Oct 07 12:28:01 op-L3-5 scylla[16180]:  [shard 0] stream_session - [Stream #61b3f2a0-650c-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=8793039, received_partitions=8117526
Oct 07 12:28:05 op-L3-5 scylla[16180]:  [shard 1] stream_session - [Stream #61b3f2a0-650c-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=8530391, received_partitions=7748081
Oct 07 12:28:06 op-L3-5 scylla[16180]:  [shard 5] stream_session - [Stream #61b3f2a0-650c-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=9450852, received_partitions=8450464
Oct 07 12:28:08 op-L3-5 scylla[16180]:  [shard 4] stream_session - [Stream #61b3f2a0-650c-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=9408408, received_partitions=7889525
Oct 07 12:29:09 op-L3-5 scylla[16180]:  [shard 4] stream_session - [Stream #69f85780-650c-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=12269394, received_partitions=8891139
Oct 07 12:29:16 op-L3-5 scylla[16180]:  [shard 1] stream_session - [Stream #69f85780-650c-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=11718867, received_partitions=8880165
Oct 07 12:29:16 op-L3-5 scylla[16180]:  [shard 0] stream_session - [Stream #69f85780-650c-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=11560253, received_partitions=8642082
Oct 07 12:29:16 op-L3-5 scylla[16180]:  [shard 2] stream_session - [Stream #69f85780-650c-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=10773484, received_partitions=8746900
Oct 07 12:29:16 op-L3-5 scylla[16180]:  [shard 5] stream_session - [Stream #69f85780-650c-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=11394480, received_partitions=8815480
Oct 07 12:29:18 op-L3-5 scylla[16180]:  [shard 3] stream_session - [Stream #69f85780-650c-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=11740169, received_partitions=9385248
Oct 07 12:30:21 op-L3-5 scylla[16180]:  [shard 5] stream_session - [Stream #f5ef55e0-650c-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=4060114, received_partitions=4075984
Oct 07 12:30:21 op-L3-5 scylla[16180]:  [shard 0] stream_session - [Stream #be594b40-650c-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=6965881, received_partitions=5976679
Oct 07 12:30:23 op-L3-5 scylla[16180]:  [shard 1] stream_session - [Stream #f5ef55e0-650c-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=4457015, received_partitions=4567775
Oct 07 12:30:26 op-L3-5 scylla[16180]:  [shard 2] stream_session - [Stream #be594b40-650c-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=7458624, received_partitions=6310283
Oct 07 12:30:28 op-L3-5 scylla[16180]:  [shard 1] stream_session - [Stream #be594b40-650c-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=6922958, received_partitions=6051176
Oct 07 12:30:29 op-L3-5 scylla[16180]:  [shard 5] stream_session - [Stream #be594b40-650c-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=7485318, received_partitions=6348268
Oct 07 12:30:30 op-L3-5 scylla[16180]:  [shard 0] stream_session - [Stream #f5ef55e0-650c-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=4618905, received_partitions=4758095
Oct 07 12:30:40 op-L3-5 scylla[16180]:  [shard 4] stream_session - [Stream #f5ef55e0-650c-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=4953325, received_partitions=4988880
Oct 07 12:30:43 op-L3-5 scylla[16180]:  [shard 2] stream_session - [Stream #f5ef55e0-650c-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=5397145, received_partitions=5440690
Oct 07 12:30:43 op-L3-5 scylla[16180]:  [shard 3] stream_session - [Stream #be594b40-650c-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=8313653, received_partitions=6848145
Oct 07 12:30:43 op-L3-5 scylla[16180]:  [shard 3] stream_session - [Stream #f5ef55e0-650c-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=5135935, received_partitions=5171373
Oct 07 12:30:44 op-L3-5 scylla[16180]:  [shard 4] stream_session - [Stream #be594b40-650c-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=8551418, received_partitions=7153562
Oct 07 12:31:23 op-L3-5 scylla[16180]:  [shard 3] stream_session - [Stream #17d9bd30-650d-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=5116497, received_partitions=4532438
Oct 07 12:31:37 op-L3-5 scylla[16180]:  [shard 0] stream_session - [Stream #17d9bd30-650d-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=5800501, received_partitions=5296394
Oct 07 12:31:38 op-L3-5 scylla[16180]:  [shard 2] stream_session - [Stream #17d9bd30-650d-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=5912528, received_partitions=4807273
Oct 07 12:31:40 op-L3-5 scylla[16180]:  [shard 1] stream_session - [Stream #17d9bd30-650d-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=5828659, received_partitions=4906344
Oct 07 12:31:40 op-L3-5 scylla[16180]:  [shard 4] stream_session - [Stream #17d9bd30-650d-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=5569791, received_partitions=4385532
Oct 07 12:31:41 op-L3-5 scylla[16180]:  [shard 5] stream_session - [Stream #17d9bd30-650d-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=6252039, received_partitions=5162002
Oct 07 12:33:48 op-L3-5 scylla[16180]:  [shard 2] stream_session - [Stream #693252f0-650d-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=4233279, received_partitions=4483519
Oct 07 12:33:58 op-L3-5 scylla[16180]:  [shard 1] stream_session - [Stream #693252f0-650d-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=4800897, received_partitions=4662915
Oct 07 12:34:12 op-L3-5 scylla[16180]:  [shard 3] stream_session - [Stream #693252f0-650d-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=5488925, received_partitions=5563885
Oct 07 12:34:22 op-L3-5 scylla[16180]:  [shard 0] stream_session - [Stream #693252f0-650d-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=5825818, received_partitions=5693924
Oct 07 12:34:26 op-L3-5 scylla[16180]:  [shard 5] stream_session - [Stream #693252f0-650d-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=6255814, received_partitions=6194640
Oct 07 12:34:31 op-L3-5 scylla[16180]:  [shard 2] stream_session - [Stream #489a8350-650d-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=9166071, received_partitions=7688146
Oct 07 12:34:31 op-L3-5 scylla[16180]:  [shard 4] stream_session - [Stream #693252f0-650d-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=6871004, received_partitions=6913505
Oct 07 12:34:43 op-L3-5 scylla[16180]:  [shard 2] stream_session - [Stream #8b684230-650d-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=6049432, received_partitions=4975610
Oct 07 12:34:44 op-L3-5 scylla[16180]:  [shard 3] stream_session - [Stream #489a8350-650d-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=9273905, received_partitions=7043621
Oct 07 12:34:47 op-L3-5 scylla[16180]:  [shard 4] stream_session - [Stream #489a8350-650d-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=9656783, received_partitions=7341631
Oct 07 12:34:47 op-L3-5 scylla[16180]:  [shard 5] stream_session - [Stream #489a8350-650d-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=9458666, received_partitions=7494794
Oct 07 12:34:47 op-L3-5 scylla[16180]:  [shard 0] stream_session - [Stream #489a8350-650d-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=10450155, received_partitions=8000800
Oct 07 12:34:51 op-L3-5 scylla[16180]:  [shard 1] stream_session - [Stream #489a8350-650d-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=10755174, received_partitions=8242347
Oct 07 12:34:59 op-L3-5 scylla[16180]:  [shard 0] stream_session - [Stream #8b684230-650d-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=5624756, received_partitions=5096406
Oct 07 12:35:05 op-L3-5 scylla[16180]:  [shard 3] stream_session - [Stream #8b684230-650d-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=6069117, received_partitions=5063640
Oct 07 12:35:05 op-L3-5 scylla[16180]:  [shard 1] stream_session - [Stream #8b684230-650d-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=6142856, received_partitions=5247660
Oct 07 12:35:11 op-L3-5 scylla[16180]:  [shard 5] stream_session - [Stream #8b684230-650d-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=6772531, received_partitions=5642146
Oct 07 12:35:13 op-L3-5 scylla[16180]:  [shard 4] stream_session - [Stream #8b684230-650d-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=6770030, received_partitions=5617650
Oct 07 12:35:46 op-L3-5 scylla[16180]:  [shard 1] stream_session - [Stream #73370570-650d-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=7240788, received_partitions=6417380
Oct 07 12:35:50 op-L3-5 scylla[16180]:  [shard 0] stream_session - [Stream #73370570-650d-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=7319946, received_partitions=6447978
Oct 07 12:35:57 op-L3-5 scylla[16180]:  [shard 2] stream_session - [Stream #73370570-650d-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=8708765, received_partitions=7691845
Oct 07 12:36:12 op-L3-5 scylla[16180]:  [shard 5] stream_session - [Stream #73370570-650d-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=8595040, received_partitions=7604757
Oct 07 12:36:14 op-L3-5 scylla[16180]:  [shard 4] stream_session - [Stream #73370570-650d-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=8915712, received_partitions=7682853
Oct 07 12:36:15 op-L3-5 scylla[16180]:  [shard 3] stream_session - [Stream #73370570-650d-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=9128377, received_partitions=7778449
Oct 07 12:36:17 op-L3-5 scylla[16180]:  [shard 2] stream_session - [Stream #f321c180-650d-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=1983493, received_partitions=2083861
Oct 07 12:36:22 op-L3-5 scylla[16180]:  [shard 1] stream_session - [Stream #f321c180-650d-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=2084775, received_partitions=1951955
Oct 07 12:36:29 op-L3-5 scylla[16180]:  [shard 3] stream_session - [Stream #f321c180-650d-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=2556894, received_partitions=2729990
Oct 07 12:36:34 op-L3-5 scylla[16180]:  [shard 5] stream_session - [Stream #f321c180-650d-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=2708509, received_partitions=2847902
Oct 07 12:36:52 op-L3-5 scylla[16180]:  [shard 0] stream_session - [Stream #f321c180-650d-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=3912423, received_partitions=2406630
Oct 07 12:36:58 op-L3-5 scylla[16180]:  [shard 4] stream_session - [Stream #f321c180-650d-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=3979780, received_partitions=2700987
Oct 07 12:38:03 op-L3-5 scylla[16180]:  [shard 5] stream_session - [Stream #3e5e2990-650e-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=1445688, received_partitions=1443009
Oct 07 12:38:09 op-L3-5 scylla[16180]:  [shard 4] stream_session - [Stream #3e5e2990-650e-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=1553933, received_partitions=1386242
Oct 07 12:38:10 op-L3-5 scylla[16180]:  [shard 0] stream_session - [Stream #3e5e2990-650e-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=1564167, received_partitions=1430528
Oct 07 12:38:12 op-L3-5 scylla[16180]:  [shard 2] stream_session - [Stream #3e5e2990-650e-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=1922449, received_partitions=1883337
Oct 07 12:38:13 op-L3-5 scylla[16180]:  [shard 3] stream_session - [Stream #3e5e2990-650e-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=1787093, received_partitions=1714181
Oct 07 12:38:14 op-L3-5 scylla[16180]:  [shard 1] stream_session - [Stream #3e5e2990-650e-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=2010898, received_partitions=1910831
Oct 07 12:38:50 op-L3-5 scylla[16180]:  [shard 5] stream_session - [Stream #0a3447d0-650e-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=6648126, received_partitions=5652548
Oct 07 12:38:55 op-L3-5 scylla[16180]:  [shard 1] stream_session - [Stream #0a3447d0-650e-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=6723605, received_partitions=5575266
Oct 07 12:38:56 op-L3-5 scylla[16180]:  [shard 4] stream_session - [Stream #0a3447d0-650e-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=6545684, received_partitions=5806482
Oct 07 12:38:56 op-L3-5 scylla[16180]:  [shard 3] stream_session - [Stream #0a3447d0-650e-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=6732574, received_partitions=5794787
Oct 07 12:38:57 op-L3-5 scylla[16180]:  [shard 0] stream_session - [Stream #0a3447d0-650e-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=6405457, received_partitions=5624842
Oct 07 12:38:59 op-L3-5 scylla[16180]:  [shard 2] stream_session - [Stream #0a3447d0-650e-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=6978471, received_partitions=5492451
Oct 07 12:39:13 op-L3-5 scylla[16180]:  [shard 2] stream_session - [Stream #086c0af0-650e-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=7065542, received_partitions=6512109
Oct 07 12:39:18 op-L3-5 scylla[16180]:  [shard 1] stream_session - [Stream #086c0af0-650e-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=7665323, received_partitions=6470564
Oct 07 12:39:22 op-L3-5 scylla[16180]:  [shard 4] stream_session - [Stream #086c0af0-650e-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=7744079, received_partitions=6334861
Oct 07 12:39:24 op-L3-5 scylla[16180]:  [shard 3] stream_session - [Stream #086c0af0-650e-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=7762211, received_partitions=6656225
Oct 07 12:39:29 op-L3-5 scylla[16180]:  [shard 0] stream_session - [Stream #086c0af0-650e-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=8224511, received_partitions=7042064
Oct 07 12:39:31 op-L3-5 scylla[16180]:  [shard 5] stream_session - [Stream #086c0af0-650e-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=8698219, received_partitions=6774672
Oct 07 12:39:50 op-L3-5 scylla[16180]:  [shard 2] stream_session - [Stream #3d9c3920-650e-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=5299016, received_partitions=4500074
Oct 07 12:40:07 op-L3-5 scylla[16180]:  [shard 4] stream_session - [Stream #3d9c3920-650e-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=5540653, received_partitions=4640761
Oct 07 12:40:12 op-L3-5 scylla[16180]:  [shard 3] stream_session - [Stream #3d9c3920-650e-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=5680347, received_partitions=4727733
Oct 07 12:40:19 op-L3-5 scylla[16180]:  [shard 1] stream_session - [Stream #3d9c3920-650e-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=5729476, received_partitions=5153112
Oct 07 12:40:20 op-L3-5 scylla[16180]:  [shard 0] stream_session - [Stream #3d9c3920-650e-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=5963382, received_partitions=5277868
Oct 07 12:40:21 op-L3-5 scylla[16180]:  [shard 5] stream_session - [Stream #3d9c3920-650e-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=5795614, received_partitions=4876626
Oct 07 12:40:23 op-L3-5 scylla[16180]:  [shard 5] stream_session - [Stream #9195df90-650e-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=1594211, received_partitions=1517526
Oct 07 12:40:29 op-L3-5 scylla[16180]:  [shard 1] stream_session - [Stream #9195df90-650e-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=1630561, received_partitions=1581434
Oct 07 12:40:30 op-L3-5 scylla[16180]:  [shard 4] stream_session - [Stream #9195df90-650e-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=1556222, received_partitions=1502099
Oct 07 12:40:30 op-L3-5 scylla[16180]:  [shard 2] stream_session - [Stream #9195df90-650e-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=1970161, received_partitions=1773333
Oct 07 12:40:31 op-L3-5 scylla[16180]:  [shard 3] stream_session - [Stream #9195df90-650e-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=1595154, received_partitions=1499443
Oct 07 12:40:32 op-L3-5 scylla[16180]:  [shard 0] stream_session - [Stream #9195df90-650e-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=1824084, received_partitions=1677226
Oct 07 12:42:59 op-L3-5 scylla[16180]:  [shard 0] stream_session - [Stream #a97811f0-650e-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=5798007, received_partitions=5181852
Oct 07 12:43:05 op-L3-5 scylla[16180]:  [shard 5] stream_session - [Stream #a97811f0-650e-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=6474002, received_partitions=5347722
Oct 07 12:43:05 op-L3-5 scylla[16180]:  [shard 4] stream_session - [Stream #a97811f0-650e-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=7461086, received_partitions=6360627
Oct 07 12:43:14 op-L3-5 scylla[16180]:  [shard 2] stream_session - [Stream #a97811f0-650e-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=6258640, received_partitions=5508326
Oct 07 12:43:15 op-L3-5 scylla[16180]:  [shard 3] stream_session - [Stream #a97811f0-650e-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=6659862, received_partitions=5775099
Oct 07 12:43:15 op-L3-5 scylla[16180]:  [shard 1] stream_session - [Stream #a97811f0-650e-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=6500936, received_partitions=5306016
Oct 07 12:44:03 op-L3-5 scylla[16180]:  [shard 5] stream_session - [Stream #c26042a0-650e-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=5308882, received_partitions=4209388
Oct 07 12:44:04 op-L3-5 scylla[16180]:  [shard 4] stream_session - [Stream #c26042a0-650e-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=5170288, received_partitions=4239976
Oct 07 12:44:06 op-L3-5 scylla[16180]:  [shard 3] stream_session - [Stream #c26042a0-650e-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=5382157, received_partitions=4381226
Oct 07 12:44:26 op-L3-5 scylla[16180]:  [shard 0] stream_session - [Stream #c26042a0-650e-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=6341862, received_partitions=5058715
Oct 07 12:44:27 op-L3-5 scylla[16180]:  [shard 2] stream_session - [Stream #c26042a0-650e-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=6513717, received_partitions=5470043
Oct 07 12:44:37 op-L3-5 scylla[16180]:  [shard 1] stream_session - [Stream #c26042a0-650e-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=7502582, received_partitions=6147006
Oct 07 12:47:03 op-L3-5 scylla[16180]:  [shard 1] stream_session - [Stream #2b30b030-650f-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=7077613, received_partitions=5663678
Oct 07 12:47:05 op-L3-5 scylla[16180]:  [shard 0] stream_session - [Stream #2b30b030-650f-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=6897936, received_partitions=5955799
Oct 07 12:47:10 op-L3-5 scylla[16180]:  [shard 2] stream_session - [Stream #2b30b030-650f-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=7087987, received_partitions=6202876
Oct 07 12:47:11 op-L3-5 scylla[16180]:  [shard 4] stream_session - [Stream #2b30b030-650f-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=7848330, received_partitions=6273369
Oct 07 12:47:12 op-L3-5 scylla[16180]:  [shard 5] stream_session - [Stream #2b30b030-650f-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=7613073, received_partitions=6099454
Oct 07 12:47:18 op-L3-5 scylla[16180]:  [shard 3] stream_session - [Stream #2b30b030-650f-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=8524357, received_partitions=6933881
Oct 07 12:47:21 op-L3-5 scylla[16180]:  [shard 1] stream_session - [Stream #5cd6cd40-650f-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=4008282, received_partitions=3687997
Oct 07 12:47:23 op-L3-5 scylla[16180]:  [shard 5] stream_session - [Stream #5cd6cd40-650f-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=3960953, received_partitions=3405625
Oct 07 12:47:31 op-L3-5 scylla[16180]:  [shard 0] stream_session - [Stream #5cd6cd40-650f-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=5005919, received_partitions=4371216
Oct 07 12:47:31 op-L3-5 scylla[16180]:  [shard 2] stream_session - [Stream #5cd6cd40-650f-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=4307390, received_partitions=3532990
Oct 07 12:47:35 op-L3-5 scylla[16180]:  [shard 4] stream_session - [Stream #5cd6cd40-650f-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=4818996, received_partitions=4025648
Oct 07 12:47:36 op-L3-5 scylla[16180]:  [shard 3] stream_session - [Stream #5cd6cd40-650f-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=4768038, received_partitions=3868311
Oct 07 12:48:18 op-L3-5 scylla[16180]:  [shard 0] stream_session - [Stream #be3ae670-650f-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=777407, received_partitions=723496
Oct 07 12:48:19 op-L3-5 scylla[16180]:  [shard 5] stream_session - [Stream #be3ae670-650f-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=692057, received_partitions=649095
Oct 07 12:48:23 op-L3-5 scylla[16180]:  [shard 2] stream_session - [Stream #be3ae670-650f-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=1018366, received_partitions=915461
Oct 07 12:48:23 op-L3-5 scylla[16180]:  [shard 3] stream_session - [Stream #be3ae670-650f-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=1017982, received_partitions=934638
Oct 07 12:48:23 op-L3-5 scylla[16180]:  [shard 1] stream_session - [Stream #be3ae670-650f-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=965872, received_partitions=845253
Oct 07 12:48:26 op-L3-5 scylla[16180]:  [shard 4] stream_session - [Stream #be3ae670-650f-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=1429222, received_partitions=1278448
Oct 07 12:48:58 op-L3-5 scylla[16180]:  [shard 1] stream_session - [Stream #c380d860-650f-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=1465732, received_partitions=1365535
Oct 07 12:48:59 op-L3-5 scylla[16180]:  [shard 0] stream_session - [Stream #c380d860-650f-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=1820586, received_partitions=1477952
Oct 07 12:49:01 op-L3-5 scylla[16180]:  [shard 3] stream_session - [Stream #c380d860-650f-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=1899010, received_partitions=1763192
Oct 07 12:49:02 op-L3-5 scylla[16180]:  [shard 5] stream_session - [Stream #c380d860-650f-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=1694978, received_partitions=1530592
Oct 07 12:49:04 op-L3-5 scylla[16180]:  [shard 4] stream_session - [Stream #c380d860-650f-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=1847046, received_partitions=1697139
Oct 07 12:49:06 op-L3-5 scylla[16180]:  [shard 2] stream_session - [Stream #c380d860-650f-11ee-ba8a-781846405d51] Write to sstable for ks=pc, cf=tidx2, estimated_partitions=2240983, received_partitions=1938889

Oct 07 12:49:19 op-L3-5 scylla[16180]:  [shard 0] storage_service - Bootstrap completed! for the tokens

Oct 07 12:55:59 op-L3-5 scylla[16180]:  [shard 4] table - Starting off-strategy compaction for pc.tidx2, 44 candidates were found
Oct 07 12:55:59 op-L3-5 scylla[16180]:  [shard 4] compaction - [Reshape pc.tidx2 dd109e40-6510-11ee-bb88-781c46405d51] Reshaping [/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-262-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-232-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-268-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-220-big-Data.db:level=0:origin=streaming]
Oct 07 12:56:01 op-L3-5 scylla[16180]:  [shard 0] table - Starting off-strategy compaction for pc.tidx2, 44 candidates were found
Oct 07 12:56:01 op-L3-5 scylla[16180]:  [shard 0] compaction - [Reshape pc.tidx2 ddfe59a0-6510-11ee-ba8a-781846405d51] Reshaping [/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-24-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-12-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-84-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-120-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-180-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-234-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-144-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-18-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-90-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-168-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-6-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-252-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-48-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-174-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-108-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-246-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-114-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-198-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-78-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-132-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-186-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-222-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-60-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-192-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-162-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-36-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-240-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-204-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-126-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-72-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-102-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-210-big-Data.db:level=0:origin=streaming]
Oct 07 12:56:13 op-L3-5 scylla[16180]:  [shard 1] table - Starting off-strategy compaction for pc.tidx2, 44 candidates were found
Oct 07 12:56:13 op-L3-5 scylla[16180]:  [shard 1] compaction - [Reshape pc.tidx2 e55b6bc0-6510-11ee-88ca-781a46405d51] Reshaping [/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-265-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-229-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-217-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-199-big-Data.db:level=0:origin=streaming]
Oct 07 12:56:18 op-L3-5 scylla[16180]:  [shard 2] table - Starting off-strategy compaction for pc.tidx2, 44 candidates were found
Oct 07 12:56:18 op-L3-5 scylla[16180]:  [shard 2] compaction - [Reshape pc.tidx2 e7ee72b0-6510-11ee-b436-781746405d51] Reshaping [/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-230-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-266-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-218-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-200-big-Data.db:level=0:origin=streaming]
Oct 07 12:56:18 op-L3-5 scylla[16180]:  [shard 5] table - Starting off-strategy compaction for pc.tidx2, 44 candidates were found
Oct 07 12:56:18 op-L3-5 scylla[16180]:  [shard 5] compaction - [Reshape pc.tidx2 e84ce660-6510-11ee-ba9f-781946405d51] Reshaping [/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-203-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-59-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-257-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-71-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-113-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-167-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-245-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-107-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-77-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-89-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-155-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-227-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-11-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-173-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-239-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-197-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-17-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-131-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-209-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-47-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-41-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-95-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-185-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-65-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-161-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-35-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-251-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-83-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-137-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-29-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-101-big-Data.db:level=0:origin=streaming]
Oct 07 12:56:22 op-L3-5 scylla[16180]:  [shard 3] table - Starting off-strategy compaction for pc.tidx2, 44 candidates were found
...
Oct 07 13:22:59 op-L3-5 scylla[16180]:  [shard 1] compaction - [Reshape pc.tidx2 e55b6bc0-6510-11ee-88ca-781a46405d51] Reshaped 32 sstables to [/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-277-big-Data.db:level=0]. 21GB to 21GB (~99% of original) in 1549583ms = 13MB/s. ~195751936 total partitions merged to 195704740.
Oct 07 13:22:59 op-L3-5 scylla[16180]:  [shard 1] compaction - [Reshape pc.tidx2 e55b6bc0-6510-11ee-88ca-781a46405d51] Reshaping [/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-277-big-Data.db:level=0:origin=reshape,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-259-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-211-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-67-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-31-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-43-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-97-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-151-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-139-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-157-big-Data.db:level=0:origin=streaming]
Oct 07 13:23:23 op-L3-5 scylla[16180]:  [shard 2] compaction - [Reshape pc.tidx2 e7ee72b0-6510-11ee-b436-781746405d51] Reshaped 32 sstables to [/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-278-big-Data.db:level=0]. 21GB to 21GB (~99% of original) in 1562733ms = 13MB/s. ~197505024 total partitions merged to 197473169.
Oct 07 13:23:23 op-L3-5 scylla[16180]:  [shard 2] compaction - [Reshape pc.tidx2 e7ee72b0-6510-11ee-b436-781746405d51] Reshaping [/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-278-big-Data.db:level=0:origin=reshape,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-146-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-260-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-68-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-44-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-98-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-170-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-152-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-140-big-Data.db:level=0:origin=streaming,/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-242-big-Data.db:level=0:origin=streaming]
Oct 07 13:52:22 op-L3-5 scylla[16180]:  [shard 5] compaction - [Reshape pc.tidx2 e84ce660-6510-11ee-ba9f-781946405d51] Reshaped 14 sstables to [/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-281-big-Data.db:level=0]. 27GB to 27GB (~99% of original) in 2005139ms = 13MB/s. ~405573632 total partitions merged to 405530827.
Oct 07 13:52:22 op-L3-5 scylla[16180]:  [shard 5] table - Done with off-strategy compaction for pc.tidx2
Oct 07 13:54:36 op-L3-5 scylla[16180]:  [shard 4] compaction - [Reshape pc.tidx2 dd109e40-6510-11ee-bb88-781c46405d51] Reshaped 10 sstables to [/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-286-big-Data.db:level=0]. 27GB to 27GB (~99% of original) in 1970191ms = 13MB/s. ~430389248 total partitions merged to 430345061.
Oct 07 13:54:36 op-L3-5 scylla[16180]:  [shard 4] table - Done with off-strategy compaction for pc.tidx2
Oct 07 13:54:46 op-L3-5 scylla[16180]:  [shard 0] compaction - [Reshape pc.tidx2 ddfe59a0-6510-11ee-ba8a-781846405d51] Reshaped 13 sstables to [/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-276-big-Data.db:level=0]. 27GB to 27GB (~99% of original) in 1963054ms = 13MB/s. ~429752320 total partitions merged to 429701825.
Oct 07 13:54:46 op-L3-5 scylla[16180]:  [shard 0] table - Done with off-strategy compaction for pc.tidx2
Oct 07 13:55:14 op-L3-5 scylla[16180]:  [shard 3] compaction - [Reshape pc.tidx2 eac25330-6510-11ee-90fa-781b46405d51] Reshaped 13 sstables to [/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-279-big-Data.db:level=0]. 27GB to 27GB (~99% of original) in 1968386ms = 13MB/s. ~427171840 total partitions merged to 427121876.
Oct 07 13:55:14 op-L3-5 scylla[16180]:  [shard 3] table - Done with off-strategy compaction for pc.tidx2
Oct 07 13:55:34 op-L3-5 scylla[16180]:  [shard 1] compaction - [Reshape pc.tidx2 e55b6bc0-6510-11ee-88ca-781a46405d51] Reshaped 10 sstables to [/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-283-big-Data.db:level=0]. 27GB to 27GB (~99% of original) in 1955395ms = 13MB/s. ~436322304 total partitions merged to 436265914.
Oct 07 13:55:34 op-L3-5 scylla[16180]:  [shard 1] table - Done with off-strategy compaction for pc.tidx2
Oct 07 13:56:15 op-L3-5 scylla[16180]:  [shard 2] compaction - [Reshape pc.tidx2 e7ee72b0-6510-11ee-b436-781746405d51] Reshaped 10 sstables to [/var/lib/scylla/data/pc/tidx2-9a443dd0a31011e9b8d6000000000000/md-284-big-Data.db:level=0]. 27GB to 27GB (~99% of original) in 1971868ms = 13MB/s. ~440545280 total partitions merged to 440502686.
Oct 07 13:56:15 op-L3-5 scylla[16180]:  [shard 2] table - Done with off-strategy compaction for pc.tidx2

(I had to shorten the log a little to fit in here)

11:59:54 - 12:49:19 - Bootstrap / stream (50 minutes)
12:49:19 - 13:56:15 - Reshape / compact (1 hour 7 minutes)

Adding 2nd node (previous log): 44 stream sessions / sstables
Adding 5th node (in the rack): 264 stream sessions / sstables

Conclusion:
With a larger number of nodes, number of sstables is growing and reshape is taking longer than streaming.

avikivity · October 9, 2023, 4:41pm

Please file an issue with all the information, it’s easier to track it there.

avikivity · October 9, 2023, 4:44pm

That’s actually expected. With a small number of nodes, the scheduler on the existing node will throttle the exiting node’s streaming bandwidth to preserve queries, so the new node’s disk bandwidth will not be saturated.

With a large number of nodes, each existing node needs to contribute a small amount of bandwidth, so we can saturate the new node. So streaming from a large cluster is faster than streaming from a small cluster.

Reshape takes place on the new node by itself, so its performance doesn’t depend on the cluster size.

horschi · October 9, 2023, 8:45pm

I was refering to the relative time between streaming & reshaping. Why should reshape have a higher ratio when the cluster is larger?
In an ideal world it should not, but I think currently its because scylla creates more&smaller sstable files when clusters are larger.

btw: I am currently bootstrapping another node (again second node in DC) and its compact like crazy during streaming:

S-1:~$ nodetool  compactionstats
pending tasks: 21
- pc.svorders2: 3
- pc.svorderids2: 4
- pc.bochangepointers: 2
- pc.tidx2: 6
- pc.boentries4: 1
- pc.svbatchids: 5

id                                   compaction type keyspace     table              completed total    unit progress
d0a6dd70-66e3-11ee-8edb-956855b9bc34 COMPACTION      pc tidx2                    17540703  22968320 keys 76.37%  
2935ea80-66e4-11ee-8edb-956855b9bc34 COMPACTION      pc svbatchids               340       512      keys 66.41%  
2ab2ed90-66e4-11ee-8edb-956855b9bc34 COMPACTION      pc svorders2                29        512      keys 5.66%   
2fc7e470-66e4-11ee-bf38-956455b9bc34 COMPACTION      pc bochangepointers         32        512      keys 6.25%   
d0b07a60-66e3-11ee-869d-956255b9bc34 COMPACTION      pc tidx2                    18346089  22747136 keys 80.65%  
29359c60-66e4-11ee-869d-956255b9bc34 COMPACTION      pc svbatchids               332       512      keys 64.84%  
2ac9d0f0-66e4-11ee-869d-956255b9bc34 COMPACTION      pc svorders2                27        512      keys 5.27%   
295420e0-66e4-11ee-b45c-956355b9bc34 COMPACTION      pc tidx2                    613262    20477952 keys 2.99%   
295a8980-66e4-11ee-b45c-956355b9bc34 COMPACTION      pc svbatchids               274       512      keys 53.52%  
2abd74e0-66e4-11ee-b45c-956355b9bc34 COMPACTION      pc svorders2                22        512      keys 4.30%   
2953f9d0-66e4-11ee-891f-956955b9bc34 COMPACTION      pc tidx2                    925716    20469760 keys 4.52%   
30751460-66e4-11ee-891f-956955b9bc34 COMPACTION      pc svorders2                0         512      keys 0.00%   
d5532770-66e3-11ee-a4a6-956655b9bc34 COMPACTION      pc tidx2                    15453415  20623360 keys 74.93%  
292e9780-66e4-11ee-a4a6-956655b9bc34 COMPACTION      pc svbatchids               327       512      keys 63.87%  
2ab389d0-66e4-11ee-a4a6-956655b9bc34 COMPACTION      pc svorders2                19        512      keys 3.71%   
d52f99e0-66e3-11ee-b678-956755b9bc34 COMPACTION      pc tidx2                    15711539  20631552 keys 76.15%  
292e9780-66e4-11ee-b678-956755b9bc34 COMPACTION      pc svbatchids               237       512      keys 46.29%  
2ac479c0-66e4-11ee-b678-956755b9bc34 COMPACTION      pc svorders2                28        512      keys 5.47%   
Active compaction remaining time :   0h00m00s

I still think bootstrapping would be faster if less sstables would be created and reshape would happen during streaming

Should I still open a ticket?