Hi i want to connect my scylladb with my managed kafka service in GCP without using confluent cloud ?, is that possible to connect
If you’re using GCPs Kafka Connect offering then according to the public documentation I’ve found it is not possible. Per the limitations listed at Kafka Connect overview | Google Cloud Managed Service for Apache Kafka :
The service does not support uploading custom connector plugins to your Kafka Connect cluster.
Which is unfortunate.
However if you could setup your own Kafka Connect cluster and configure it so that it works with brokers used by your Managed Kafka Service then it should be doable. In such case you could run any custom connector on your Connect cluster, which includes GitHub - scylladb/scylla-cdc-source-connector: A Kafka source connector capturing Scylla CDC changes and GitHub - scylladb/kafka-connect-scylladb: Kafka Connect Scylladb Sink
One is a CDC connector capable of reading CDC log tables and streaming them to Kafka and another one is a connector reading from Kafka topics and inserting messages into ScyllaDB tables. Before trying to set anything up make sure that you are not hindered by their limitations. Not every complex data type or mode is currently supported.
1 Like