Running a cluster with Docker Compose

ScyllaDB
,
1

Hello everyone!

I’m trying to deploy a ScyllaDB cluster using docker-compose. I have the following YAML file:

name: scylla-cluster

services:

  scylla-node-1:
    container_name: scylla-node-1
    image: scylladb/scylla:latest
    command:
      - --seeds=scylla-node-1,scylla-node-2,scylla-node-3
      - --smp=1
      - --memory=2G
      - --overprovisioned=1
      - --api-address=0.0.0.0

  scylla-node-2:
    container_name: scylla-node-2
    image: scylladb/scylla:latest
    command:
      - --seeds=scylla-node-1,scylla-node-2,scylla-node-3
      - --smp=1
      - --memory=2G
      - --overprovisioned=1
      - --api-address=0.0.0.0

  scylla-node-3:
    container_name: scylla-node-3
    image: scylladb/scylla:latest
    command:
      - --seeds=scylla-node-1,scylla-node-2,scylla-node-3
      - --smp=1
      - --memory=2G
      - --overprovisioned=1
      - --api-address=0.0.0.0

When I run docker-compose up, I get the following error from each node:

FATAL: Exception during startup, aborting: std::runtime_error (Could not setup Async I/O: Resource temporarily unavailable. The required nr_event (1) exceeds the limit of request capacity in /proc/sys/fs/aio-max-nr (65536). Try increasing that number or reducing the amount of logical CPUs available for your application)

Now, if I run cat /proc/sys/fs/aio-max-nr on the host machine, I get 1048576. If I run the same command inside one of the containers, I get 65536.What should I do to increase the aio-max-nr inside the container? Or what should I do to make it work in general?

This is what running lscpu gives me inside a container:

Architecture:             x86_64
  CPU op-mode(s):         32-bit, 64-bit
  Address sizes:          48 bits physical, 48 bits virtual
  Byte Order:             Little Endian
CPU(s):                   8
  On-line CPU(s) list:    0-7
Vendor ID:                AuthenticAMD
  Model name:             AMD Ryzen 5 PRO 6650U with Radeon Graphics
    CPU family:           25
    Model:                68
    Thread(s) per core:   1
    Core(s) per socket:   8
    Socket(s):            1
    Stepping:             1
    BogoMIPS:             5789.06
    Flags:                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm rep_good nopl cpuid extd_apicid tsc_known_freq pni p
                          clmulqdq ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw perfctr_c
                          ore ssbd ibrs ibpb stibp vmmcall fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr wbnoinvd arat npt lbrv n
                          rip_save tsc_scale vmcb_clean pausefilter pfthreshold v_vmsave_vmload vgif umip pku vaes vpclmulqdq rdpid fsrm arch_capabilities
Virtualization features:  
  Virtualization:         AMD-V
  Hypervisor vendor:      KVM
  Virtualization type:    full
Caches (sum of all):      
  L1d:                    512 KiB (8 instances)
  L1i:                    512 KiB (8 instances)
  L2:                     4 MiB (8 instances)
  L3:                     128 MiB (8 instances)
Vulnerabilities:          
  Gather data sampling:   Not affected
  Itlb multihit:          Not affected
  L1tf:                   Not affected
  Mds:                    Not affected
  Meltdown:               Not affected
  Mmio stale data:        Not affected
  Reg file data sampling: Not affected
  Retbleed:               Not affected
  Spec rstack overflow:   Vulnerable: Safe RET, no microcode
  Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:             Mitigation; Retpolines; IBPB conditional; IBRS_FW; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
  Srbds:                  Not affected
  Tsx async abort:        Not affected
2

Apparently, the issue was that I was using Docker Desktop for Linux, that runs inside a VM instead of the host OS.

This meant that my containerized /proc tree was different than my host OS /proc.

I tried to search of a way to edit the /proc of the VM but did not find anything, so I just installed regular docker engine on the host and tried it that way, and it finally worked:

Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address    Load      Tokens Owns Host ID                              Rack 
UN 172.18.0.2 347.61 KB 256    ?    143bb70e-47c0-47de-b2f3-bfc127652952 rack1
UN 172.18.0.3 364.01 KB 256    ?    295509cb-1d3e-4132-836d-a5a8c6c95b40 rack1
UN 172.18.0.4 309.18 KB 256    ?    695fa35f-3f89-4fb4-8303-52cc7d88f0de rack1

Note: Non-system keyspaces don't have the same replication settings, effective ownership information is meaningless
1 Like