Load and Stream issue with Memory Allocation

Hey guys,

Running into issues while trying to load one of the tables. I’ve retried it multiple times by dropping the table and still getting same error. Not sure what is causing the issue. Other table works fine but this one table is giving us an issue.

Ubuntu 22.04
24 CPU 
502GB RAM 
scylla --version 5.2.9-0.20230920.5709d0043978

This is the error I’m getting

Nov 13 16:25:26 scylladb-02 scylla[3870]:  [shard 14] storage_proxy - Exception when communicating with 10.31.13.102, to read from system.batchlog: std::bad_alloc (std::bad_alloc)
Nov 13 16:25:26 scylladb-02 scylla[3870]:  [shard  0] batchlog_manager - Exception in batch replay: exceptions::read_failure_exception (Operation failed for system.batchlog - received 0 responses and 1 failures from 1 CL=ONE.)
Nov 13 16:25:26 scylladb-02 scylla[3870]:  [shard  0] migration_manager - Requesting schema pull from 10.31.2.110:0
Nov 13 16:25:26 scylladb-02 scylla[3870]:  [shard  0] migration_manager - Pulling schema from 10.31.2.110:0
Nov 13 16:25:26 scylladb-02 scylla[3870]:  [shard  0] schema_tables - Schema version changed to f36da9c7-de1c-3909-a119-08f591baa675
Nov 13 16:25:26 scylladb-02 scylla[3870]:  [shard  0] migration_manager - Schema merge with 10.31.2.110:0 completed
Nov 13 16:25:26 scylladb-02 scylla[3870]:  [shard  0] mutation_reader - shard_reader::close(): read_ahead on shard 46 failed: std::bad_alloc (std::bad_alloc)
Nov 13 16:25:27 scylladb-02 scylla[3870]:  [shard  0] mutation_reader - shard_reader::close(): read_ahead on shard 46 failed: std::bad_alloc (std::bad_alloc)
Nov 13 16:25:27 scylladb-02 scylla[3870]:  [shard  0] mutation_reader - shard_reader::close(): read_ahead on shard 46 failed: std::bad_alloc (std::bad_alloc)
Nov 13 16:25:27 scylladb-02 scylla[3870]:  [shard  0] mutation_reader - shard_reader::close(): read_ahead on shard 46 failed: std::bad_alloc (std::bad_alloc)
Nov 13 16:25:27 scylladb-02 scylla[3870]:  [shard  0] storage_proxy - Exception when communicating with 10.31.13.102, to read from system_distributed.service_levels: std::bad_alloc (std::bad_alloc)
Nov 13 16:25:34 scylladb-02 scylla[3870]:  [shard  0] mutation_reader - shard_reader::close(): read_ahead on shard 46 failed: std::bad_alloc (std::bad_alloc)
Nov 13 16:25:34 scylladb-02 scylla[3870]:  [shard  0] mutation_reader - shard_reader::close(): read_ahead on shard 46 failed: std::bad_alloc (std::bad_alloc)
Nov 13 16:25:37 scylladb-02 scylla[3870]:  [shard  0] migration_manager - Requesting schema pull from 10.31.2.110:0
Nov 13 16:25:37 scylladb-02 scylla[3870]:  [shard  0] migration_manager - Pulling schema from 10.31.2.110:0
Nov 13 16:25:37 scylladb-02 scylla[3870]:  [shard  0] schema_tables - Schema version changed to f36da9c7-de1c-3909-a119-08f591baa675
Nov 13 16:25:37 scylladb-02 scylla[3870]:  [shard  0] migration_manager - Schema merge with 10.31.2.110:0 completed
Nov 13 16:25:37 scylladb-02 scylla[3870]:  [shard  0] mutation_reader - shard_reader::close(): read_ahead on shard 46 failed: std::bad_alloc (std::bad_alloc)
Nov 13 16:25:37 scylladb-02 scylla[3870]:  [shard  0] mutation_reader - shard_reader::close(): read_ahead on shard 46 failed: std::bad_alloc (std::bad_alloc)
Nov 13 16:25:37 scylladb-02 scylla[3870]:  [shard  0] mutation_reader - shard_reader::close(): read_ahead on shard 46 failed: std::bad_alloc (std::bad_alloc)
Nov 13 16:25:37 scylladb-02 scylla[3870]:  [shard  0] mutation_reader - shard_reader::close(): read_ahead on shard 46 failed: std::bad_alloc (std::bad_alloc)
Nov 13 16:25:37 scylladb-02 scylla[3870]:  [shard  0] storage_proxy - Exception when communicating with 10.31.13.102, to read from system_distributed.service_levels: std::bad_alloc (std::bad_alloc)

Interesting. 5.2.9 does include a fix for a similar problem sstableloader/nodetool refresh: bad_alloc (seastar - Failed to allocate 536870912 bytes) · Issue #13491 · scylladb/scylladb · GitHub . Can you open a GitHub issue instead? And then please provide full logs. :slight_smile: