Hi ScyllaDB team,
For instance monitoring we use Dynatrace Oneagent software. This software needs to start before ScyllaDB starts. So we install OneAgent, starte OneAgent service then ScyllaDB.
But Scylla OSS 5.2.9 fails to start and dumps below log. Can someone help resolving this error please.
systemd-coredump: Process 29201 (scylla) of user 991 dumped core.#012#012Stack trace of thread 29201:#012#0 0x00007f4baf32f2a0 n/a (liboneagentproc.so)#012#1 0x0000000004fb1cda _Unwind_RaiseException (scylla)#012#2 0x00007f4bada4674b __cxa_throw (libstdc++.so.6)#012#3 0x00007f4bada3a772 __cxa_guard_acquire.cold (libstdc++.so.6)#012#4 0x0000000004fb1c67 dl_iterate_phdr (scylla)#012#5 0x00007f4baf340a66 n/a (liboneagentproc.so)#012#6 0x00007f4baf346eaa n/a (liboneagentproc.so)#012#7 0x00007f4baf347130 n/a (liboneagentproc.so)#012#8 0x00007f4baf3437d4 dlsym (liboneagentproc.so)#012#9 0x0000000004fb1c7c dl_iterate_phdr (scylla)#012#10 0x00007f4baf340a66 n/a (liboneagentproc.so)#012#11 0x00007f4baf346eaa n/a (liboneagentproc.so)#012#12 0x00007f4baf347130 n/a (liboneagentproc.so)#012#13 0x00007f4baf3426cc fopen (liboneagentproc.so)#012#14 0x00007f4baeb96603 _gnutls_fips_mode_enabled (libgnutls.so.30)#012#15 0x00007f4baeb8630a _gnutls_rnd_preinit (libgnutls.so.30)#012#16 0x00007f4baeb7615d _gnutls_global_init (libgnutls.so.30)#012#17 0x00007f4baeb44fa3 lib_init (libgnutls.so.30)#012#18 0x00007f4baf548cde call_init (ld.so)#012#19 0x00007f4baf548dcc _dl_init (ld.so)#012#20 0x00007f4baf55f8e0 n/a (ld.so)
Thanks,
Swaroop
We have seen problems with Dynatrace agent running alongside Scylla before. As it hooks to processes, it can cause not only Scylla to stall but in worse scenarios it may even cause the process to get hang. The general recommendation is simply not to run 3rd party processes along with Scylla. But if you really really must it, then contact your Dynatrace support team and ask for instructions on how to tell the agent not to hook to the Scylla process and also for the proper way to pin the agent to use specific CPUs, that Scylla is/will not actively (be) using.
Newest version of Dynatrace OneAgent was fixed and now coexist with Scylla > 5.2.
â—Ź oneagent.service - Dynatrace OneAgent
Loaded: loaded (/etc/systemd/system/oneagent.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2024-06-21 10:28:03 CEST; 4min 49s ago
Process: 1065 ExecStart=/opt/dynatrace/oneagent/agent/initscripts/oneagent start (code=exited, status=0/SUCCESS)
Main PID: 1178 (oneagentwatchdo)
Tasks: 50 (limit: 38413)
Memory: 244.8M
CGroup: /system.slice/oneagent.service
├─1178 /opt/dynatrace/oneagent/agent/lib64/oneagentwatchdog -bg -config=/opt/dynatrace/oneagent/agent/conf/watchdog.conf
├─1187 oneagentos -Dcom.compuware.apm.WatchDogTimeout=900 -watchdog.restart_file_location=/var/lib/dynatrace/oneagent/a>
├─1203 oneagentplugin -Dcom.compuware.apm.WatchDogTimeout=900 -Dcom.compuware.apm.WatchDogPort=50001
├─1205 oneagentloganalytics -Dcom.compuware.apm.WatchDogTimeout=900 -Dcom.compuware.apm.WatchDogPort=50002
├─1207 oneagentnetwork -Dcom.compuware.apm.WatchDogTimeout=900 -Dcom.compuware.apm.WatchDogPort=50003
├─1273 /opt/dynatrace/oneagent/agent/lib64/oneagenteventstracer --logdir /var/log/dynatrace/oneagent/os --cputimeoffset>
└─1958 /opt/dynatrace/oneagent/agent/lib64/oneagentebpfdiscovery --log-dir /var/log/dynatrace/oneagent/os/ --log-no-std>
Jun 21 10:28:03 systemd[1]: Starting Dynatrace OneAgent…
Jun 21 10:28:03 oneagent[1182]: 10:28:03 Dynatrace OneAgent service started.
Jun 21 10:28:03 systemd[1]: Started Dynatrace OneAgent.
â—Ź scylla-server.service - Scylla Server
Loaded: loaded (/lib/systemd/system/scylla-server.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/scylla-server.service.d
└─capabilities.conf, dependencies.conf, sysconfdir.conf
Active: active (running) since Fri 2024-06-21 10:28:14 CEST; 4min 46s ago
Process: 861 ExecStartPre=/opt/scylladb/scripts/scylla_prepare (code=exited, status=0/SUCCESS)
Main PID: 1069 (scylla)
Status: “serving”
Tasks: 8 (limit: 38413)
Memory: 29.2G
CGroup: /scylla.slice/scylla-server.slice/scylla-server.service
└─1069 /usr/bin/scylla --log-to-syslog 1 --log-to-stdout 0 --default-log-level info --network-stack posix --io-properti>
Jun 21 10:28:39 scylla[1069]: [shard 0:main] init - Scylla version 5.4.7-0.20240602.98139a8716ab initia
1 Like
Thanks for the update. But we have upgraded to 6.0.0 and we don’t see this issue anymore