# Repair by token ranges **URL:** \ **Category:** ScyllaDB\ **Tags:** administration, repair\ **Created:** [July 8, 2025, 7:22am UTC](https://forum.scylladb.com/t/repair-by-token-ranges/4964 "2025-07-08T07:22:06Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Mark.B](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.scylladb.com/mark.b/32/297_2.png) [@Mark.B](https://forum.scylladb.com/u/Mark.B)\ **Post date:** [July 8, 2025, 7:22am UTC](https://forum.scylladb.com/t/repair-by-token-ranges/4964/1 "2025-07-08T07:22:07Z")
Hello. I’m trying to use repair API with token ranges and have a number of questions. 1. Empty [start | end]Token: Consider the following result: ```auto curl 10.0.32.70:10000/storage_service/range_to_endpoint_map/markb | jq -r '.[] | select (.key[0] == "" or .key[1] == "")' { "key": [ "9218948411409126567", "" ], "value": [ "10.0.32.70", "10.0.32.74", "10.0.32.77", "10.0.32.78", "10.0.32.75", "10.0.32.71" ] } { "key": [ "", "-9213378514646091305" ], "value": [ "10.0.32.70", "10.0.32.74", "10.0.32.77", "10.0.32.78", "10.0.32.75", "10.0.32.71" ] } ``` The following command returns: `{"message": "boost::wrapexcept (bad lexical cast: source type value could not be interpreted as target)", "code": 500}`. ```auto curl -X POST '10.0.32.70:10000/storage_service/repair_async/markb?ranges=9218948411409126567%3A' ``` Q1.1: Is it possible to repair this token range using the `ranges` parameter? Q1.2: If A1.1 is “no”, then what’s the proper command to repair such a token range (if I need to repair it at all)? 1. Consider the following result: ```auto curl 10.0.32.70:10000/storage_service/range_to_endpoint_map/markb | jq -r '.[] | select (.key[0] == "-1569604604390479271")' { "key": [ "-1569604604390479271", "-1519586698886497919" ], "value": [ "10.0.32.70", "10.0.32.75", "10.0.32.72", "10.0.32.78", "10.0.32.76", "10.0.32.74" ] } ``` Notice, that there is no 10.0.32.71 host in the endpoint list of this token range. ```auto curl -X POST 'http://10.0.32.71:10000/storage_service/repair_async/markb?primaryRange=true&ranges=-1569604604390479271%3A-1519586698886497919' ``` But the command above “successfully” repairs this token range according to the result of the following command (with X returned by the above command). The result doesn’t change, if I don’t use the primaryRange parameter. ```auto curl 'http://10.0.32.71:10000/storage_service/repair_status/?id=X' ``` Q2: How is it possible to successfully run a repair command of a token range on a host which doesn’t present in the endpoint list of this token range? Does this command really repairs a token range on other hosts, when a repair master host doesn’t have any replica of this data? Thanks in advance. ---
**Author:** ![Guy](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.scylladb.com/guy/32/9_2.png) [@Guy](https://forum.scylladb.com/u/Guy)\ **Post date:** [July 9, 2025, 3:36am UTC](https://forum.scylladb.com/t/repair-by-token-ranges/4964/2 "2025-07-09T03:36:12Z")
Please share more info about your deployment (ScyllaDB Version, OS, etc.) ---
**Author:** ![nopzdk](https://avatars.discourse-cdn.com/v4/letter/n/dec6dc/32.png) [@nopzdk](https://forum.scylladb.com/u/nopzdk)\ **Post date:** [July 9, 2025, 7:46am UTC](https://forum.scylladb.com/t/repair-by-token-ranges/4964/3 "2025-07-09T07:46:03Z")
Why don’t you use Scylla Manager to manage cluster repair and backup? ---
**Author:** ![Mark.B](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.scylladb.com/mark.b/32/297_2.png) [@Mark.B](https://forum.scylladb.com/u/Mark.B)\ **Post date:** [July 9, 2025, 10:26am UTC](https://forum.scylladb.com/t/repair-by-token-ranges/4964/4 "2025-07-09T10:26:05Z")
It’s a test system. Scylladb 5.4.4 Community Debian 10 3 DC x 3 nodes each The reason I cat’t use Scylla Manager is its 5-node limit for open-source editions. But I believe, that results of these API calls are the same on all Scylla versions. BTW, the `nodetool repair [-pr] -st X -et Y keyspace` commands return SUCCESSFUL as well on whatever X, Y values disregarding the host where I run it. For example, the following command finishes “successfully” as well. ```auto nodetool repair -pr -st 0 -et -1 keyspace ``` So, if all such commands return “success”, it would be good to know if they really do some useful work. And if not, that I need to know the correct algorithm of my cluster reparation by token ranges. I’m not able to construct such an algorithm based on available documentation at the moment, so I need some help here. Lets presume, that I need to repair a keyspace `markb` in my cluster. My algorithm with not clear enough info is below. 1. Get range (start\_token, end\_token) to endpoint mapping with `storage_service/range_to_endpoint_map` API (like in Q2 of the initial post). 2. Run the following command (or its API analogue with the same parameters) for all (start\_token, end\_token) pairs on the 1-st host of the endpoint list. ```auto ssh 10.0.32.7x -- nodetool repair [-pr] -st -et markb ``` I believe, that a (start\_token, end\_token) pair must be provided to the `nodetool` utility as is from the `storage_service/range_to_endpoint_map` API call (both return/accept such an interval as exclusive-inclusive, so I don’t need any ±1 math here). It’s not clear here what to do with token ranges returned by `storage_service/range_to_endpoint_map` without start\_token or end\_token. Should I ignore these ranges? Should I run some special repair command on them? It’s not clear if I need to use the `-pr` parameter here. Seems, that it should be no difference, because `10.0.32.70` holds a primary range and I run the command namely on this host. I must (or could?) omit `-pr` running this command on, say, `10.0.32.74`, because it holds replica of this token range. Please, correct me if it’s not a correct algorithm to repair a keyspace by token ranges. ---
**Author:** ![Mark.B](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.scylladb.com/mark.b/32/297_2.png) [@Mark.B](https://forum.scylladb.com/u/Mark.B)\ **Post date:** [July 10, 2025, 5:01pm UTC](https://forum.scylladb.com/t/repair-by-token-ranges/4964/5 "2025-07-10T17:01:15Z")
Seems, that the ScyllaDB’s implementation of `nodetool repair` with -st / -et is not compatible with the Cassandra’s one. ScyllaDB doesn’t check parameters at all. Below is a couple of examples which must finish with an error as on, say, Cassandra 4.1.9. 1. The host doesn’t have any replica of this token range: ```auto # ssh 10.202.110.142 -- nodetool repair -st -3389892500827521034 -et -3359020880985253656 markb [2025-07-10 16:38:15,028] Starting repair command #3 (46aff4a0-5dac-11f0-ac52-7d59c9ea8df9), repairing keyspace markb with repair options (parallelism: parallel, primary range: false, incremental: true, job threads: 1, ColumnFamilies: [], dataCenters: [], hosts: [], previewKind: NONE, # of ranges: 1, pull repair: false, force repair: false, optimise streams: false, ignore unreplicated keyspaces: false, repairPaxos: true, paxosOnly: false) [2025-07-10 16:38:15,028] Repair command #3 failed with error Nothing to repair for (-3389892500827521034,-3359020880985253656] in markb - aborting [2025-07-10 16:38:15,029] Repair command #3 finished with error error: Repair job has failed with the error message: Repair command #3 failed with error Nothing to repair for (-3389892500827521034,-3359020880985253656] in markb - aborting. Check the logs on the repair participants for further details -- StackTrace -- java.lang.RuntimeException: Repair job has failed with the error message: Repair command #3 failed with error Nothing to repair for (-3389892500827521034,-3359020880985253656] in markb - aborting. Check the logs on the repair participants for further details at org.apache.cassandra.tools.RepairRunner.progress(RepairRunner.java:137) at org.apache.cassandra.utils.progress.jmx.JMXNotificationProgressListener.handleNotification(JMXNotificationProgressListener.java:77) at java.management/com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.dispatchNotification(ClientNotifForwarder.java:633) at java.management/com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(ClientNotifForwarder.java:555) at java.management/com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:474) at java.management/com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor.lambda$execute$0(ClientNotifForwarder.java:108) at java.base/java.lang.Thread.run(Thread.java:829) ``` 1. Wrong token range boundaries: ```auto # ssh 10.202.110.144 -- nodetool repair -st -3359020880985253656 -et -3312316956446460988 markb [2025-07-10 16:42:34,644] Starting repair command #5 (e16e84c0-5dac-11f0-9606-23fd3a216128), repairing keyspace markb with repair options (parallelism: parallel, primary range: false, incremental: true, job threads: 1, ColumnFamilies: [], dataCenters: [], hosts: [], previewKind: NONE, # of ranges: 1, pull repair: false, force repair: false, optimise streams: false, ignore unreplicated keyspaces: false, repairPaxos: true, paxosOnly: false) [2025-07-10 16:42:34,645] Repair command #5 failed with error Requested range (-3359020880985253656,-3312316956446460988] intersects a local range ((-3359020880985253656,-3335108523467758869]) but is not fully contained in one; this would lead to imprecise repair. keyspace: markb [2025-07-10 16:42:34,646] Repair command #5 finished with error error: Repair job has failed with the error message: Repair command #5 failed with error Requested range (-3359020880985253656,-3312316956446460988] intersects a local range ((-3359020880985253656,-3335108523467758869]) but is not fully contained in one; this would lead to imprecise repair. keyspace: markb. Check the logs on the repair participants for further details -- StackTrace -- java.lang.RuntimeException: Repair job has failed with the error message: Repair command #5 failed with error Requested range (-3359020880985253656,-3312316956446460988] intersects a local range ((-3359020880985253656,-3335108523467758869]) but is not fully contained in one; this would lead to imprecise repair. keyspace: markb. Check the logs on the repair participants for further details at org.apache.cassandra.tools.RepairRunner.progress(RepairRunner.java:137) at org.apache.cassandra.utils.progress.jmx.JMXNotificationProgressListener.handleNotification(JMXNotificationProgressListener.java:77) at java.management/com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.dispatchNotification(ClientNotifForwarder.java:633) at java.management/com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(ClientNotifForwarder.java:555) at java.management/com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:474) at java.management/com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor.lambda$execute$0(ClientNotifForwarder.java:108) at java.base/java.lang.Thread.run(Thread.java:829) ``` ScyllaDB returns “SUCCESSFUL” in both cases, which is incorrect in my opinion. P.S.: The same behavior is on ScyllaDB 6.2.3 Community… ---
**Author:** ![felipemendes](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.scylladb.com/felipemendes/32/11_2.png) [@felipemendes](https://forum.scylladb.com/u/felipemendes)\ **Post date:** [July 11, 2025, 3:08pm UTC](https://forum.scylladb.com/t/repair-by-token-ranges/4964/6 "2025-07-11T15:08:29Z")
As explained under a [different venue](https://scylladb-users.slack.com/archives/C2NLNBXLN/p1752242910601649?thread_ts=1751909603.345299&cid=C2NLNBXLN) the _SUCCESSFUL_ message indicates the non-replica node works only as a repair coordinator. In fact, if you shutdown one of the actual replicas involved in the repair, then its status should change to _FAILED_. With regards to tokens missing a `` or ``, this indicates the range wrap arounds on `/range_to_endpoint_map`. > [@Mark.B](#): > > ScyllaDB returns “SUCCESSFUL” in both cases, which is incorrect in my opinion. That said, it is arguable whether the behavior your observe is incorrect or not, as repair is taking place and thus fulfilling its goal, though not in most efficient way. Forcing it to only accept a replica coordinator could be an option, warning the user about it another, or simply adding a boolean flag yet another. Note that both `/range_to_endpoint_map/{keyspace}` as well as (my favorite) `/describe_ring/{keyspace}` include the relevant endpoints owning a particular range. That said, incorrectly triggering repair to a non-replica is unlikely to manifest in practice unless manually forced by the user as your example demonstrates. Please file a [GitHub issue](https://github.com/scylladb/scylladb/issues) and explain your use case should our APIs be missing on anything. ---
**Author:** ![Mark.B](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.scylladb.com/mark.b/32/297_2.png) [@Mark.B](https://forum.scylladb.com/u/Mark.B)\ **Post date:** [July 11, 2025, 3:42pm UTC](https://forum.scylladb.com/t/repair-by-token-ranges/4964/7 "2025-07-11T15:42:34Z")
> [@felipemendes](#): > > the _SUCCESSFUL_ message indicates the non-replica node works only as a repair coordinator It becomes more interesting 🙂 Did I get it right, that Scylla `nodetool repair` implementation can run on non-replica node to repair a full token range? This means, that it’s possible to run it on a single node to repair the whole cluster. Is that correct? Well, it’s cool enough, but it should be clearly documented somewhere. We all know, that it’s impossible to do such things in Cassandra, Datastax. Even ScyllaDB docs say, that we must run `nodetool repair` on all cluster hosts! I’ll definitely open an issue on this, if I have time. Either with a request to clarify this incompatible behavior (for example, to clearly state, that a successful run on a non-replica node really repairs the corresponding range and not silently swallows the call doing nothing useful as one could expect), or to forbid such calls… ---
**Author:** ![felipemendes](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.scylladb.com/felipemendes/32/11_2.png) [@felipemendes](https://forum.scylladb.com/u/felipemendes)\ **Post date:** [July 11, 2025, 4:05pm UTC](https://forum.scylladb.com/t/repair-by-token-ranges/4964/8 "2025-07-11T16:05:41Z")
> [@Mark.B](#): > > Did I get it right, that Scylla `nodetool repair` implementation can run on non-replica node to repair a full token range? This means, that it’s possible to run it on a single node to repair the whole cluster. Is that correct? When you specify a token range to repair, yes. This alone, is a very particular use case, which on its own already require retrieving the token-to-replica mapping on its own. So the subsequent repair will often get executed on a replica node anyway. You may check the `scylla-server` journalctl logs as you invoke it on a non-replica node and compare it versus a replica node. In the former case, all replicas plus the non-replica node will be involved during the repair. In the latter, only the actual replicas, hence why the latter is more efficient. I don’t remember the logic around `nodetool repair` alone (`-pr` isn’t subject to this as it already assumed the primary ranges of the particular invoked replica), but if mind serves me well ScyllaDB would distribute the tasks accordingly across the natural endpoints. But best to check logs to confirm for a relatively sparse table. ---
**Author:** ![Guy](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.scylladb.com/guy/32/9_2.png) [@Guy](https://forum.scylladb.com/u/Guy)\ **Post date:** [July 14, 2025, 3:18pm UTC](https://forum.scylladb.com/t/repair-by-token-ranges/4964/9 "2025-07-14T15:18:30Z")
And here is the Slack thread for reference: > [@](#): > > [**Originally from the User Slack**](https://slack.scylladb.com/) > > ![Mark_Barinstein](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/0/0fa8041f379df858db2a82792f8af99a5f9f1a49.png) **@Mark\_Barinstein:** Hello. > > I’m trying to use repair API with token ranges and have a number of questions. > > 1. Empty [start | end]Token: > Consider the following result: > > ```auto > curl 10.0.32.70:10000/storage_service/range_to_endpoint_map/markb | jq -r '.[] | select (.key[0] == "" or .key[1] == "")' > > { > "key": [ > "9218948411409126567", > "" > ], > "value": [ > "10.0.32.70", > "10.0.32.74", > "10.0.32.77", > "10.0.32.78", > "10.0.32.75", > "10.0.32.71" > ] > } > { > "key": [ > "", > "-9213378514646091305" > ], > "value": [ > "10.0.32.70", > "10.0.32.74", > "10.0.32.77", > "10.0.32.78", > "10.0.32.75", > "10.0.32.71" > ] > } > > ``` > > The following command returns `{"message": "boost::wrapexcept (bad lexical cast: source type value could not be interpreted as target)", "code": 500}` > > ```auto > curl -X POST '10.0.32.70:10000/storage_service/repair_async/markb?ranges=9218948411409126567%3A' > > ``` > > Q1.1: Is it possible to repair this token range using the `ranges` parameter? > Q1.2: If A1.1 is “no”, then what’s the proper command to repair such a token range (if I need to repair it at all)? > > 1. Consider the following result: > > ```auto > curl 10.0.32.70:10000/storage_service/range_to_endpoint_map/markb | jq -r '.[] | select (.key[0] == "-1569604604390479271")' > > { > "key": [ > "-1569604604390479271", > "-1519586698886497919" > ], > "value": [ > "10.0.32.70", > "10.0.32.75", > "10.0.32.72", > "10.0.32.78", > "10.0.32.76", > "10.0.32.74" > ] > } > > ``` > > Notice, that there is no `10.0.32.71` in the endpoints list of this token range. > > ```auto > curl -X POST '' > > ``` > > But the command above “successfully” repairs this token range according to the result of the following command (with X returned by the above command). > The result doesn’t change, if I don’t use the `primaryRange` parameter. > > ```auto > curl '' > > ``` > > Q2: How is it possible to successfully run a repair command of a token range on a host which doesn’t present in the endpoints list of this token range? > Is it some “feature”? > > Thanks in advance. > FYI: > > ```auto > nodetool repair [-pr] -st -1569604604390479271 -et -1519586698886497919 markb > > ``` > > The command above behaves the same: it always returns success disregarding the host where I run it. > Moreover, I even get success if I specify **whatever** start & end tokens like: > > ```auto > nodetool repair -st 0 -et -1 markb > > ``` > > How could one trust such a result? > > ![Felipe_Cardeneti_Mendes](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/1X/233a9d524fbc97e39f08633c16b3dd2575d1d1f4.jpeg) **@Felipe\_Cardeneti\_Mendes:** well, your first `repair_async` command doesn’t contain an ending token, so clearly it fails. The former `range_to_endpoint_map` output means it wraps around, so you should apply the relevant values instead. > > Personally, I find `/storage_service/describe_ring/{keyspace}` more convenient. Consider: > > ```auto > { > "start_token": "9160570400458636044", > "end_token": "-9185903984411353280", > "endpoints": [ > "172.31.13.187", > "172.31.3.5" > ], > "rpc_endpoints": [ > "172.31.13.187", > "172.31.3.5" > ], > "endpoint_details": [ > { > "host": "172.31.13.187", > "datacenter": "datacenter1", > "rack": "rack2" > }, > { > "host": "172.31.3.5", > "datacenter": "datacenter1", > "rack": "rack3" > } > ] > } > > ``` > > And the following list of operations: > > ```auto > root@ip-172-31-4-30:/var/lib/scylla# curl -X POST '127.0.0.1:10000/storage_service/repair_async/system_traces?ranges=9160570400458636044:-9185903984411353280' > 6 > root@ip-172-31-4-30:/var/lib/scylla# curl 127.0.0.1:10000/storage_service/repair_async/system_auth?id=6 > "SUCCESSFUL" > > ``` > > `172.31.4.30` isn’t a replica and received the request to repair the range owned by 2 other replicas. In that sense, it is a repair coordinator, and you’ll observe the result of this coordination under its logs. > > That said, you may call it a “feature”, but as you can imagine it is a bit pointless to start a repair task using a non-replica coordinator, as you’ll transfer data around unnecessarily. > > Overall, Scylla Manager streamlines all this logic, including when tablets are used which have their own APIs for repairing. > > ![Mark_Barinstein](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/0/0fa8041f379df858db2a82792f8af99a5f9f1a49.png) **@Mark\_Barinstein:** Indeed, the `storage_service/describe_ring` API result doesn’t have such a “feature” with empty key boundary unlike `storage_service/range_to_endpoint_map`: > > ```auto > $ curl -s ${host?}:10000/storage_service/describe_ring/${ks?} | jq -r ".[] | select ( .start_token == \"\" or .end_token == \"\" )" > $ > > $ curl -s ${host?}:10000/storage_service/describe_ring/${ks?} | jq -r ".[] | select ( (.start_token | tonumber) > (.end_token | tonumber) ) | {"start_token": .start_token, "endtoken": .end_token, "endpoints": .endpoints} " > { > "start_token": "9218948411409126567", > "endtoken": "-9213378514646091305", > "endpoints": [ > "10.0.32.70", > "10.0.32.74", > "10.0.32.77", > "10.0.32.78", > "10.0.32.75", > "10.0.32.71" > ] > } > $ > > ``` > > Will use `describe_ring` instead. I started to use `range_to_endpoint_map` because of its smaller result set - w/o useless fields in my case. > Thanks! > > BTW, more problems with the Scylla `nodetool repair` utility (and similar API). > Its result is not compatible with Cassandra’s one sometimes. > [https://forum.scylladb.com/t/repair-by-token-ranges/4964](https://forum.scylladb.com/t/repair-by-token-ranges/4964) > > > ScyllaDB Community NoSQL Forum: Repair by token ranges ---
**Author:** ![Mark.B](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.scylladb.com/mark.b/32/297_2.png) [@Mark.B](https://forum.scylladb.com/u/Mark.B)\ **Post date:** [July 16, 2025, 11:20am UTC](https://forum.scylladb.com/t/repair-by-token-ranges/4964/10 "2025-07-16T11:20:03Z")
> [@felipemendes](#): > > When you specify a token range to repair, yes. This alone, is a very particular use case, which on its own already require retrieving the token-to-replica mapping on its own. So the subsequent repair will often get executed on a replica node anyway. Q1: Do I get it right, that such a non-replica node becomes a “replay master” node anyway, and this node can’t participate in any other parallel repairs until the end of this one, even this non-replica node doesn’t hold any data being repaired? ```auto $ curl -s ${host?}:10000/storage_service/describe_ring/${ks?} | jq -r ".[] | select ( (.start_token | tonumber) > (.end_token | tonumber) ) | {"start_token": .start_token, "endtoken": .end_token, "endpoints": .endpoints} " { "start_token": "9218948411409126567", "endtoken": "-9213378514646091305", "endpoints": [ "10.0.32.70", "10.0.32.74", "10.0.32.77", "10.0.32.78", "10.0.32.75", "10.0.32.71" ] } ``` Q2: Do I get it right that it’s some “fake” token range above (with start\_token \> end\_token) which can’t hold any data, and we can omit its reparation?