Cant start on VPS that does enough memory?

ScyllaDB
, ,
1

I am starting scylla for on a VPS for a very small project, when I configure with 1 cpu and 500M or even 750M scylla refuses to start, even with --cpuset 1 setup and with/without --overprovisioned

scylla[5761]: command used: "/usr/bin/scylla --log-to-syslog 1 --log-to-stdout 0 --default-log-level info --network-stack posix --smp 1 --memory 550M --io-properties-file=/etc/scylla.d/io>
scylla[5761]: pid: 5761 scylla[5761]: parsed command line options: [log-to-syslog, (positional) 1, log-to-stdout, (positional) 0, default-log-level, (positional) info, network-stack, (positional) posix, smp, (po> 
scylla[5761]: Could not initialize seastar: std::runtime_error (insufficient physical memory: needed 524288000 available 500000000)

As far as I know the VPS should have enough memory:

               total        used        free      shared  buff/cache   available
Mem:           1.9Gi       332Mi       513Mi       2.3Mi       1.3Gi       1.6Gi
Swap:          1.0Gi          0B       1.0Gi

Im still trying to poke around and work out why, but I thought I would post here in the mean time to see if anyone has some useful pointers in the right direction for me. I’ve never had this problem with Cassandra before.

After some googling I have discovered that this will increase the reported “free” memory, but scylla still will not start:

sync; echo 3 > /proc/sys/vm/drop_caches
free -h
               total        used        free      shared  buff/cache   available
Mem:           1.9Gi       310Mi       1.6Gi       2.3Mi       188Mi       1.6Gi
Swap:          1.0Gi          0B       1.0Gi

There still appears to be some strange hard limit somewhere:

scylla[11074]: command used: "/usr/bin/scylla --log-to-syslog 1 --log-to-stdout 0 --default-log-level info --network-stack posix --smp 1 --memory 560M --io-properties-file=/etc/scylla.d/i>
scylla[11074]: Could not initialize seastar: std::runtime_error (insufficient physical memory: needed 534773760 available 500000000)
2

Seastar will not consider swap when calculating available memory, it will only consider RAM. Furthermore, when considering the vailable RAM, it will substract the kernel reserve, which it reads from /proc/sys/vm/min_free_kbytes. Looks like on your system this amount is 500 MB, which is less them the minimal amount seastar was told to allocate for itself (--memory parameter).

See seastar/src/core/resource.cc at bd27a45289ec13fddddcf383436cd6141a32c1b0 · scylladb/seastar · GitHub for the full calculation.

1 Like