# MultiDC cluster ,Observing 100% cpu when connecting scylla with kafka source connector to read cdc logs

**URL:** <https://forum.scylladb.com/t/multidc-cluster-observing-100-cpu-when-connecting-scylla-with-kafka-source-connector-to-read-cdc-logs/1276>\
**Category:** ScyllaDB\
**Tags:** troubleshooting, cdc, kafka, multi-dc\
**Created:** [February 5, 2024, 8:00pm UTC](https://forum.scylladb.com/t/multidc-cluster-observing-100-cpu-when-connecting-scylla-with-kafka-source-connector-to-read-cdc-logs/1276 "2024-02-05T20:00:45Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![amitesh88](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.scylladb.com/amitesh88/32/638_2.png) [@amitesh88](https://forum.scylladb.com/u/amitesh88)\
**Post date:** [February 5, 2024, 8:00pm UTC](https://forum.scylladb.com/t/multidc-cluster-observing-100-cpu-when-connecting-scylla-with-kafka-source-connector-to-read-cdc-logs/1276/1 "2024-02-05T20:00:45Z")

</div>

We have our multi dc setup with 3 node in dc1 and 3 in dc2 . Although both the DCs are in the same region n same subnet, This is done as we require separate clusters for reading n writing data.

Our setup creates a new table everyday at 12 midnight with cdc enabled in DC1

At the same time we also create a kafka source connector to consume cdc logs from DC2 everyday

Issue:  
At around 12  
When creating a new source con , we observe scylladb servers on dc2 consumes 100% cpu.

We increased cpu from 16cores to 32 but still same behavior  
Once the kafka connector creates its topic and start reading data from cdc log ,scylladb cpu cools down

Logs:

The logs in syslog shows reader\_concurrency\_semaphores for that time period.

Any expert thoughts is appreciated  
Thanks in advance

---

<div class="post-metadata">

**Author:** ![roy](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.scylladb.com/roy/32/576_2.png) [@roy](https://forum.scylladb.com/u/roy)\
**Post date:** [February 12, 2024, 1:10am UTC](https://forum.scylladb.com/t/multidc-cluster-observing-100-cpu-when-connecting-scylla-with-kafka-source-connector-to-read-cdc-logs/1276/2 "2024-02-12T01:10:36Z")

</div>

The cpu utilization itself isn’t always an indication of a problem, however the reader\_concurrency\_semaphore may be a stronger indication.

For how long do you have this high utlization and the reader\_conccurency\_semaphore errors?

Also, how exactly are your CDC configured?

Full logs would be more helpful, but sounds like a better place for it would be a Github issue.

---

<div class="post-metadata">

**Author:** ![amitesh88](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.scylladb.com/amitesh88/32/638_2.png) [@amitesh88](https://forum.scylladb.com/u/amitesh88)\
**Post date:** [February 12, 2024, 2:09am UTC](https://forum.scylladb.com/t/multidc-cluster-observing-100-cpu-when-connecting-scylla-with-kafka-source-connector-to-read-cdc-logs/1276/3 "2024-02-12T02:09:36Z")

</div>

Thanks a lot for responding

Already raised this on github

Here is the link ,along with logs and other necessary details you need

> <https://github.com/scylladb/scylla-cdc-source-connector/issues/47#issuecomment-1937481042>
>
> We have our multi dc setup with 3 node in dc1 and 3 in dc2 . Although both the D…Cs are in the same region n same subnet, This is done as we require separate clusters for reading n writing data.
> 
> Our setup creates a new table everyday at 12 midnight with cdc enabled in DC1
> 
> At the same time we also create a kafka source connector to consume cdc logs from DC2 everyday
> 
> Issue:
> At around 12
> When creating a new source con , we observe scylladb servers on dc2 consumes 100% cpu.
> 
> We increased cpu from 16cores to 32 but still same behavior
> Once the kafka connector creates its topic and start reading data from cdc log ,scylladb cpu cools down
> 
> Logs:
> 
> The logs in syslog shows reader\_concurrency\_semaphores for that time period.
> 
> Any expert thoughts is appreciated
> Thanks in advance

The reader\_concurreny\_semaphore log only comes for 15 mins when a new kafka debezium source connector spawns. Cpu also goes high during that time .  
Exact details are available in the link, please go  
through it and help in understandng the exact issue

---

<div class="post-metadata">

**Author:** ![amitesh88](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.scylladb.com/amitesh88/32/638_2.png) [@amitesh88](https://forum.scylladb.com/u/amitesh88)\
**Post date:** [February 12, 2024, 6:50pm UTC](https://forum.scylladb.com/t/multidc-cluster-observing-100-cpu-when-connecting-scylla-with-kafka-source-connector-to-read-cdc-logs/1276/4 "2024-02-12T18:50:33Z")

</div>

We manage to reduce 15 mins of High CPUby tweaking **scylla.query.time.window.size** in kafka source connector , but still no clue about 100% cpu utilisation

Also it is observed , our 6 node dc2 cluster and 4 nodes kafka connector, there is a total of 40000 active connection on each scylladb node

Any clue about this??
