# Facing errors in repair jobs

**URL:** <https://forum.scylladb.com/t/facing-errors-in-repair-jobs/2293>\
**Category:** ScyllaDB\
**Tags:** troubleshooting, repair\
**Created:** [July 12, 2024, 11:19am UTC](https://forum.scylladb.com/t/facing-errors-in-repair-jobs/2293 "2024-07-12T11:19:50Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![ChaitanyaT](https://avatars.discourse-cdn.com/v4/letter/c/d9b06d/32.png) [@ChaitanyaT](https://forum.scylladb.com/u/ChaitanyaT)\
**Post date:** [July 12, 2024, 11:19am UTC](https://forum.scylladb.com/t/facing-errors-in-repair-jobs/2293/1 "2024-07-12T11:19:50Z")

</div>

Getting below error in logs while repair job. This is causing repair job to run for long hours.  
Error :

Jul 9 05:55:58 nodename scylla: [shard 25] repair - repair[f06d168e-664f-4b3f-9717-5abc0e4cbf01]: shard=25, keyspace=time\_series\_interaction, cf=click\_logged\_in, range=(-7693897  
20434297988, -551877587955844925], got error in row level repair: std::runtime\_error (put\_row\_diff: Repair follower=10.x.x.x failed in put\_row\_diff hanlder, status=0)  
Jul 9 05:55:58 nodename scylla: [shard 28] repair - repair[f06d168e-664f-4b3f-9717-5abc0e4cbf01]: shard=28, keyspace=time\_series\_interaction, cf=click\_logged\_in, range=(-7693897  
20434297988, -551877587955844925], got error in row level repair: std::runtime\_error (put\_row\_diff: Repair follower=10.x.x.x failed in put\_row\_diff hanlder, status=0)  
Jul 9 05:55:58 nodename scylla: [shard 11] repair - repair[f06d168e-664f-4b3f-9717-5abc0e4cbf01]: shard=11, keyspace=time\_series\_interaction, cf=click\_logged\_in, range=(-7693897

Jul 9 06:04:41 nodename scylla: [shard 0] repair - repair[f06d168e-664f-4b3f-9717-5abc0e4cbf01]: repair\_tracker run failed: std::runtime\_error ({shard 0: seastar::rpc::closed\_e  
rror (connection is closed), shard 1: seastar::rpc::closed\_error (connection is closed), shard 2: seastar::rpc::closed\_error (connection is closed), shard 3: seastar::rpc::closed\_error (connection i  
s closed), shard 4: seastar::rpc::closed\_error (connection is closed), shard 5: seastar::rpc::remote\_verb\_error (seastar::nested\_exception), shard 6: seastar::rpc::closed\_error (connection is closed  
), shard 7: seastar::rpc::closed\_error (connection is closed), shard 8: seastar::rpc::closed\_error (connection is closed), shard 9: seastar::rpc::closed\_error (connection is closed), shard 10: seast  
ar::rpc::closed\_error (connection is closed), shard 11: seastar::rpc::remote\_verb\_error (seastar::nested\_exception), shard 12: seastar::rpc::closed\_error (connection is closed), shard 13: seastar::r  
pc::closed\_error (connection is closed), shard 14: seastar::rpc::closed\_error (connection is closed), shard 15: seastar::rpc::closed\_error (connection is closed), shard 16: seastar::rpc::closed\_erro

---

<div class="post-metadata">

**Author:** ![Guy](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.scylladb.com/guy/32/9_2.png) [@Guy](https://forum.scylladb.com/u/Guy)\
**Post date:** [July 14, 2024, 10:47am UTC](https://forum.scylladb.com/t/facing-errors-in-repair-jobs/2293/2 "2024-07-14T10:47:11Z")

</div>

Please share as many details as possible: ScyllaDB Version, cluster size, OS, hardware details, cluster size.  
Did you check the Monitoring dashboards?

---

<div class="post-metadata">

**Author:** ![ChaitanyaT](https://avatars.discourse-cdn.com/v4/letter/c/d9b06d/32.png) [@ChaitanyaT](https://forum.scylladb.com/u/ChaitanyaT)\
**Post date:** [July 15, 2024, 6:55pm UTC](https://forum.scylladb.com/t/facing-errors-in-repair-jobs/2293/3 "2024-07-15T18:55:33Z")

</div>

ScyllaDB version : 5.2.14  
Cluster size : 1.5 TB with 2 nodes and RF as 2.  
OS : ubuntu 20.04 focal  
Hardware : n2-highmem-32 GCP with 16 NVME disks  
Monitoring dashboards seems showing spikes in latencies in read and write.

---

<div class="post-metadata">

**Author:** ![Botond\_Denes](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.scylladb.com/botond_denes/32/163_2.png) [@Botond\_Denes](https://forum.scylladb.com/u/Botond_Denes)\
**Post date:** [July 17, 2024, 5:03am UTC](https://forum.scylladb.com/t/facing-errors-in-repair-jobs/2293/4 "2024-07-17T05:03:14Z")

</div>

It looks like the error happened on one of the peer nodes. Please check the logs of the other nodes participating in the repair, to see the real error.

---

<div class="post-metadata">

**Author:** ![ChaitanyaT](https://avatars.discourse-cdn.com/v4/letter/c/d9b06d/32.png) [@ChaitanyaT](https://forum.scylladb.com/u/ChaitanyaT)\
**Post date:** [July 17, 2024, 9:59am UTC](https://forum.scylladb.com/t/facing-errors-in-repair-jobs/2293/5 "2024-07-17T09:59:27Z")

</div>

> [@ChaitanyaT](#):
>
> error in row level repair

Errors :  
Jul 12 01:53:54 nodename scylla: [shard 2] repair - repair[b3fbc952-171d-4420-9a82-fb7981a32d9f]: Started to repair 1 out of 1 tables in keyspace=, table=, table\_id=c195a300-baaa-11ee-b6  
9d-54c54cd21d49, repair\_reason=repair  
Jul 12 01:53:54 nodename scylla: [shard 17] repair - repair[b3fbc952-171d-4420-9a82-fb7981a32d9f]: Started to repair 1 out of 1 tables in keyspace=, table=, table\_id=c195a300-baaa-11ee-b6  
9d-54c54cd21d49, repair\_reason=repair  
Jul 12 01:53:54 nodename scylla: [shard 28] repair - repair[b3fbc952-171d-4420-9a82-fb7981a32d9f]: Started to repair 1 out of 1 tables in keyspace=, table=, table\_id=c195a300-baaa-11ee-b6  
9d-54c54cd21d49, repair\_reason=repair  
Jul 12 01:53:54 nodename scylla: [shard 5] repair - repair[b3fbc952-171d-4420-9a82-fb7981a32d9f]: Started to repair 1 out of 1 tables in keyspace=, table=, table\_id=c195a300-baaa-11ee-b6  
9d-54c54cd21d49, repair\_reason=repair  
Jul 12 01:53:54 nodename scylla: [shard 20] repair - repair[b3fbc952-171d-4420-9a82-fb7981a32d9f]: Started to repair 1 out of 1 tables in keyspace=, table=, table\_id=c195a300-baaa-11ee-b6  
9d-54c54cd21d49, repair\_reason=repair  
Jul 12 01:53:54 nodename scylla: [shard 9] repair - repair[b3fbc952-171d-4420-9a82-fb7981a32d9f]: Started to repair 1 out of 1 tables in keyspace=, table=, table\_id=c195a300-baaa-11ee-b6  
9d-54c54cd21d49, repair\_reason=repair  
Jul 12 01:53:58 nodename scylla: [shard 24] storage\_proxy - Exception when communicating with 10.x.x.x, to read from .click\_anon: std::bad\_alloc  
Jul 12 01:53:59 nodename scylla: [shard 17] compaction - [Compact system.compaction\_history 9b0d8f00-3ff1-11ef-ae9c-8bda48236624] Compacting [/var/lib/scylla/data/system/compaction\_history-b4dbb7b4dc493fb5b3bfce6e434832ca/me-  
183887-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/system/compaction\_history-b4dbb7b4dc493fb5b3bfce6e434832ca/me-183857-big-Data.db:level=0:origin=compaction]  
Jul 12 01:53:59 nodename scylla: [shard 17] compaction - [Compact system.compaction\_history 9b0d8f00-3ff1-11ef-ae9c-8bda48236624] Compacted 2 sstables to [/var/lib/scylla/data/system/compaction\_history-b4dbb7b4dc493fb5b3bfce6  
e434832ca/me-183917-big-Data.db:level=0]. 299kB to 192kB (~64% of original) in 107ms = 2MB/s. ~1024 total partitions merged to 844.  
Jul 12 01:53:59 nodename scylla: [shard 7] repair - repair[b3fbc952-171d-4420-9a82-fb7981a32d9f]: shard=7, keyspace=, cf=, range=(-2136115943339218610, -2031250356567146432], got error i  
n row level repair: seastar::rpc::remote\_verb\_error (std::bad\_alloc)  
Jul 12 01:54:00 nodename scylla: [shard 20] repair - repair[b3fbc952-171d-4420-9a82-fb7981a32d9f]: shard=20, keyspace=, cf=, range=(-2136115943339218610, -2031250356567146432], got error in row level repair: seastar::rpc::remote\_verb\_error (std::bad\_alloc)  
Jul 12 01:54:00 nodename scylla: [shard 3] storage\_proxy - Exception when communicating with 10.x.x.x, to read from .: std::bad\_alloc  
Jul 12 01:54:00 nodename scylla: [shard 3] storage\_proxy - Exception when communicating with 10.x.x.x, to read from .: std::bad\_alloc  
Jul 12 01:54:00 nodename scylla: [shard 18] storage\_proxy - Exception when communicating with 10.x.x.x, to read from .: std::bad\_alloc  
Jul 12 01:54:00 nodename scylla: [shard 27] storage\_proxy - Exception when communicating with 10.x.x.x, to read from .: std::bad\_alloc  
Jul 12 01:54:00 nodename scylla: [shard 18] storage\_proxy - Exception when communicating with 10.x.x.x, to read from .: std::bad\_alloc  
Jul 12 01:54:00 nodename scylla: [shard 20] repair - repair[b3fbc952-171d-4420-9a82-fb7981a32d9f]: shard=20, keyspace=, cf=, range=(-2141427493649988622, -2136115943339218610], got error in row level repair: seastar::rpc::remote\_verb\_error (std::bad\_alloc)  
Jul 12 01:54:00 nodename scylla: [shard 18] storage\_proxy - Exception when communicating with 10.x.x.x, to read from .: std::bad\_alloc  
Jul 12 01:54:00 nodename scylla: [shard 3] storage\_proxy - Exception when communicating with 10.x.x.x, to read from .: std::bad\_alloc  
Jul 12 01:54:00 nodename scylla: [shard 18] storage\_proxy - Exception when communicating with 10.x.x.x, to read from .: std::bad\_alloc  
Jul 12 01:54:00 nodename scylla: [shard 26] storage\_proxy - Exception when communicating with 10.x.x.x, to read from .: std::bad\_alloc  
Jul 12 01:54:01 nodename scylla: [shard 18] storage\_proxy - Exception when communicating with 10.x.x.x, to read from .: std::bad\_alloc  
Jul 12 01:54:01 nodename scylla: [shard 27] storage\_proxy - Exception when communicating with 10.x.x.x, to read from .: std::bad\_alloc  
Jul 12 01:54:01 nodename scylla: [shard 26] storage\_proxy - Exception when communicating with 10.x.x.x, to read from .: std::bad\_alloc  
Jul 12 01:54:01 nodename scylla: [shard 26] storage\_proxy - Exception when communicating with 10.x.x.x, to read from .: std::bad\_alloc  
Jul 12 01:54:01 nodename scylla: [shard 3] repair - repair[b3fbc952-171d-4420-9a82-fb7981a32d9f]: shard=3, keyspace=, cf=, range=(-2141427493649988622, -2136115943339218610], got error in row level repair: seastar::rpc::remote\_verb\_error (std::bad\_alloc)`  
Jul 12 01:54:01 nodename scylla: [shard 3] repair - repair[b3fbc952-171d-4420-9a82-fb7981a32d9f]: shard=3, keyspace=, cf=, range=(-2136115943339218610, -2031250356567146432], got error in row level repair: seastar::rpc::remote\_verb\_error (std::bad\_alloc)  
Jul 12 01:54:02 nodename scylla: [shard 10] repair - repair[b3fbc952-171d-4420-9a82-fb7981a32d9f]: shard=10, keyspace=, cf=, range=(-2141427493649988622, -2136115943339218610], got error in row level repair: seastar::rpc::remote\_verb\_error (std::bad\_alloc)  
Jul 12 01:54:02 nodename scylla: [shard 4] storage\_proxy - Exception when communicating with 10.x.x.x, to read from .: std::bad\_alloc  
Jul 12 01:54:02 nodename scylla: [shard 26] storage\_proxy - Exception when communicating with 10.x.x.x, to read from .: std::bad\_alloc  
Jul 12 01:54:02 nodename scylla: [shard 26] storage\_proxy - Exception when communicating with 10.x.x.x, to read from .: std::bad\_alloc  
Jul 12 01:54:02 nodename scylla: [shard 27] storage\_proxy - Exception when communicating with 10.x.x.x, to read from .: std::bad\_alloc  
Jul 12 01:54:02 nodename scylla: [shard 23] repair - repair[b3fbc952-171d-4420-9a82-fb7981a32d9f]: shard=23, keyspace=, cf=, range=(-2141427493649988622, -2136115943339218610], got error in row level repair: seastar::rpc::remote\_verb\_error (std::bad\_alloc)  
Jul 12 01:54:03 nodename scylla: [shard 0] repair - repair[b3fbc952-171d-4420-9a82-fb7981a32d9f]: shard=0, keyspace=, cf=, range=(-2136115943339218610, -2031250356567146432], got error in row level repair: std::runtime\_error (put\_row\_diff: Repair follower=10.x.x.x failed in put\_row\_diff hanlder, status=0)  
Jul 12 01:54:03 nodename scylla: [shard 18] storage\_proxy - Exception when communicating with 10.x.x.x, to read from .: std::bad\_alloc  
Jul 12 01:54:03 nodename scylla: [shard 27] storage\_proxy - Exception when communicating with 10.x.x.x, to read from .: std::bad\_alloc  
Jul 12 01:54:03 nodename scylla: message repeated 2 times: [[shard 27] storage\_proxy - Exception when communicating with 10.x.x.x, to read from .: std::bad\_alloc]  
Jul 12 01:54:03 nodename scylla: [shard 24] storage\_proxy - Exception when communicating with 10.x.x.x, to read from .click\_anon: std::bad\_alloc  
Jul 12 01:54:03 nodename scylla: [shard 17] compaction - [Compact .click\_anon 9db30c80-3ff1-11ef-ae9c-8bda48236624] Compacting [/var/lib/scylla/data//click\_anon-c576fbe0baaa11eeb69d54c54cd21d49/me-588917-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data//click\_anon-c576fbe0baaa11eeb69d54c54cd21d49/me-588887-big-Data.db:level=0:origin=compaction]  
Jul 12 01:54:03 nodename scylla: [shard 17] compaction - [Compact . 9dba1160-3ff1-11ef-ae9c-8bda48236624] Compacting [/var/lib/scylla/data//-bdb00460baaa11eeb69d54c54cd21d49/me-577157-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data//-bdb00460baaa11eeb69d54c54cd21d49/me-577127-big-Data.db:level=0:origin=compaction]

Also we tried changing repair job intensity from 1 to 2 and then 4, resulted in restart of scylla-server service.

---

<div class="post-metadata">

**Author:** ![Botond\_Denes](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.scylladb.com/botond_denes/32/163_2.png) [@Botond\_Denes](https://forum.scylladb.com/u/Botond_Denes)\
**Post date:** [July 17, 2024, 11:14am UTC](https://forum.scylladb.com/t/facing-errors-in-repair-jobs/2293/6 "2024-07-17T11:14:58Z")

</div>

I see `std::bad_alloc`:s in the logs. This is the reason repair failed.  
Increasing repair intensity will make this problem worse, that is why you had a restart (crash).

Check the `Bloom Filter memory usage (percentage)` graph, on the `Detailed` dashboard in monitoring. What does it show?

---

<div class="post-metadata">

**Author:** ![ChaitanyaT](https://avatars.discourse-cdn.com/v4/letter/c/d9b06d/32.png) [@ChaitanyaT](https://forum.scylladb.com/u/ChaitanyaT)\
**Post date:** [July 18, 2024, 8:27am UTC](https://forum.scylladb.com/t/facing-errors-in-repair-jobs/2293/7 "2024-07-18T08:27:55Z")

</div>

![Screenshot 2024-07-18 at 1.57.18 PM](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/d/d3a29edfb68e16c60bfed6dd611552216cf868eb.png)  
spikes is when repair was running.

---

<div class="post-metadata">

**Author:** ![Botond\_Denes](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.scylladb.com/botond_denes/32/163_2.png) [@Botond\_Denes](https://forum.scylladb.com/u/Botond_Denes)\
**Post date:** [July 18, 2024, 9:33am UTC](https://forum.scylladb.com/t/facing-errors-in-repair-jobs/2293/8 "2024-07-18T09:33:52Z")

</div>

There is a [fix](https://github.com/scylladb/scylladb/commit/065f7178ab) for this in 5.2.18. Upgrade to either the latest 5.2, or better, to 5.4 or 6.0 (5.2 is not supported anymore).

---

<div class="post-metadata">

**Author:** ![ChaitanyaT](https://avatars.discourse-cdn.com/v4/letter/c/d9b06d/32.png) [@ChaitanyaT](https://forum.scylladb.com/u/ChaitanyaT)\
**Post date:** [July 18, 2024, 1:04pm UTC](https://forum.scylladb.com/t/facing-errors-in-repair-jobs/2293/9 "2024-07-18T13:04:57Z")

</div>

So you are saying its because of a bug which has hit on 5.2.14 ?

---

<div class="post-metadata">

**Author:** ![Botond\_Denes](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.scylladb.com/botond_denes/32/163_2.png) [@Botond\_Denes](https://forum.scylladb.com/u/Botond_Denes)\
**Post date:** [July 22, 2024, 7:12am UTC](https://forum.scylladb.com/t/facing-errors-in-repair-jobs/2293/10 "2024-07-22T07:12:35Z")

</div>

The bug was not introduced in 5.2.14 (as far as I remember), but it was fixed in 5.2.18.

---

<div class="post-metadata">

**Author:** ![ChaitanyaT](https://avatars.discourse-cdn.com/v4/letter/c/d9b06d/32.png) [@ChaitanyaT](https://forum.scylladb.com/u/ChaitanyaT)\
**Post date:** [July 24, 2024, 7:15am UTC](https://forum.scylladb.com/t/facing-errors-in-repair-jobs/2293/11 "2024-07-24T07:15:30Z")

</div>

Thanks Botond\_Denes. This was helpful.