Illustration Image

What happened after I create Storage Attached Index in Cassandra 5?

How did I create SAI

CREATE INDEX sai__my_table__my_column ON MY_KEYSPACE.my_table(my_column) USING 'sai';

SAI in being constructed(part of logs)

Some 'Maximum memory usage reached' are logged

INFO  [Native-Transport-Requests-1] 2025-05-26 09:06:29,234 ColumnFamilyStore.java:1052 - Enqueuing flush of system_schema.indexes, Reason: INTERNALLY_FORCED, Usage: 340B (0%) on-heap, 193B (0%) off-heap
INFO  [PerDiskMemtableFlushWriter_0:22] 2025-05-26 09:06:29,252 Flushing.java:153 - Writing Memtable-indexes@327963785(171B serialized bytes, 1 ops, 340B (0%) on-heap, 193B (0%) off-heap), flushed range = [min(-9223372036854775808), max(9223372036854775807))
INFO  [PerDiskMemtableFlushWriter_0:22] 2025-05-26 09:06:29,253 Flushing.java:179 - Completed flushing /bitnami/cassandra/data/data/system_schema/indexes-0feb57ac311f382fba6d9024d305702f/da-3gql_0pat_1ekzk2rp94k38bttl9-bti-Data.db (139B) for commitlog position CommitLogPosition(segmentId=1748224867994, position=6623148)
INFO  [Native-Transport-Requests-1] 2025-05-26 09:06:29,290 ColumnFamilyStore.java:1052 - Enqueuing flush of system_schema.tables, Reason: INTERNALLY_FORCED, Usage: 924B (0%) on-heap, 861B (0%) off-heap
INFO  [PerDiskMemtableFlushWriter_0:23] 2025-05-26 09:06:29,304 Flushing.java:153 - Writing Memtable-tables@49436128(824B serialized bytes, 1 ops, 924B (0%) on-heap, 861B (0%) off-heap), flushed range = [min(-9223372036854775808), max(9223372036854775807))
INFO  [PerDiskMemtableFlushWriter_0:22] 2025-05-26 09:06:29,348 Flushing.java:153 - Writing Memtable-keyspaces@491501704(154B serialized bytes, 1 ops, 340B (0%) on-heap, 171B (0%) off-heap), flushed range = [min(-9223372036854775808), max(9223372036854775807))
INFO  [PerDiskMemtableFlushWriter_0:22] 2025-05-26 09:06:29,349 Flushing.java:179 - Completed flushing /bitnami/cassandra/data/data/system_schema/keyspaces-abac5682dea631c5b535b3d6cffd0fb6/da-3gql_0pat_208be2rp94k38bttl9-bti-Data.db (118B) for commitlog position CommitLogPosition(segmentId=1748224867994, position=6623148)
INFO  [Native-Transport-Requests-1] 2025-05-26 09:06:29,395 SecondaryIndexManager.java:226 - Index [sai__my_table__my_column] registered and writable.
INFO  [Native-Transport-Requests-1] 2025-05-26 09:06:29,396 ColumnFamilyStore.java:1052 - Enqueuing flush of my_keyspace.my_table, Reason: SCHEMA_CHANGE, Usage: 0B (0%) on-heap, 0B (0%) off-heap
INFO  [SecondaryIndexManagement:1] 2025-05-26 09:06:29,407 StorageAttachedIndex.java:874 - [my_keyspace.my_table.sai__my_table__my_column] Submitting 2 parallel initial index builds over 23 total sstables...

INFO  [SecondaryIndexExecutor:2] 2025-05-26 09:06:29,421 V1OnDiskFormat.java:161 - [my_keyspace.my_table.sai__my_table__my_column] Starting a compaction index build. Global segment memory usage: 0B
INFO  [SecondaryIndexExecutor:1] 2025-05-26 09:06:29,421 V1OnDiskFormat.java:161 - [my_keyspace.my_table.sai__my_table__my_column] Starting a compaction index build. Global segment memory usage: 0B
INFO  [SecondaryIndexExecutor:2] 2025-05-26 09:12:21,382 V1OnDiskFormat.java:161 - [my_keyspace.my_table.sai__my_table__my_column] Starting a compaction index build. Global segment memory usage: 63.323MiB
INFO  [SecondaryIndexExecutor:2] 2025-05-26 09:16:17,676 NoSpamLogger.java:104 - Maximum memory usage reached (1.000GiB) for chunk-cache buffer pool, cannot allocate chunk of 8.000MiB
INFO  [Messaging-EventLoop-3-9] 2025-05-26 09:16:55,623 OutboundConnection.java:1153 - cass-prod-test-cassandra-0/192.168.183.228:7000(/192.168.183.228:50412)->cass-prod-test-cassandra-0/192.168.183.228:7000-SMALL_MESSAGES-1e70a6b3 successfully connected, version = 13, framing = CRC, encryption = unencrypted
INFO  [Messaging-EventLoop-3-12] 2025-05-26 09:16:55,623 InboundConnectionInitiator.java:503 - /192.168.183.228:7000(/192.168.183.228:50412)->cass-prod-test-cassandra-0/192.168.183.228:7000-SMALL_MESSAGES-2a3dbced messaging connection established, version = 13, framing = CRC, encryption = unencrypted
INFO  [SecondaryIndexExecutor:2] 2025-05-26 09:31:17,676 NoSpamLogger.java:104 - Maximum memory usage reached (1.000GiB) for chunk-cache buffer pool, cannot allocate chunk of 8.000MiB
INFO  [SecondaryIndexExecutor:2] 2025-05-26 09:45:13,734 V1OnDiskFormat.java:161 - [my_keyspace.my_table.sai__my_table__my_column] Starting a compaction index build. Global segment memory usage: 396.754MiB

AFTER 3 HOURS

From the logs, I thought the SAI is created within 3 hours.

INFO  [SecondaryIndexManagement:1] 2025-05-26 12:14:04,544 SecondaryIndexManager.java:1860 - Index [sai__my_table__my_column] became queryable after successful build.

However, the table is compacted again after 12 HOURS

This is time when I am already able to query the table with SAI

INFO  [CompactionExecutor:660] 2025-05-27 01:09:23,044 CompactionTask.java:167 - Compacting (39aa7330-3a97-11f0-b61d-d569d536760d) [/bitnami/cassandra/data/data/my_keyspace/my_table-5113370028ff11ef910297a7d29eadf0/nb-29945-big-Data.db:level=0, ]
INFO  [CompactionExecutor:660] 2025-05-27 01:09:23,060 V1OnDiskFormat.java:161 - [my_keyspace.my_table.sai__my_table__my_column] Starting a compaction index build. Global segment memory usage: 0B
INFO  [NonPeriodicTasks:1] 2025-05-27 01:14:03,135 BigFormat.java:231 - Deleting sstable: /bitnami/cassandra/data/data/my_keyspace/my_table-5113370028ff11ef910297a7d29eadf0/nb-29945-big

My questions

  1. How to handle logs 'Maximum memory usage reached'? What metric from the cassandra.yaml should I increase, the file_cache_size ? If I increased the metric, will it be a problem or bring extra burden to the database?
  2. The compaction seems promising, but why there are logs about the compaction and new SAI again in 12 hours, when the SAI is already queryable and writeable?
  3. Any suggestion to shorten the SAI creation process with some workaround?
Become part of our
growing community!
Welcome to Planet Cassandra, a community for Apache Cassandra®! We're a passionate and dedicated group of users, developers, and enthusiasts who are working together to make Cassandra the best it can be. Whether you're just getting started with Cassandra or you're an experienced user, there's a place for you in our community.
A dinosaur
Planet Cassandra is a service for the Apache Cassandra® user community to share with each other. From tutorials and guides, to discussions and updates, we're here to help you get the most out of Cassandra. Connect with us and become part of our growing community today.
© 2009-2023 The Apache Software Foundation under the terms of the Apache License 2.0. Apache, the Apache feather logo, Apache Cassandra, Cassandra, and the Cassandra logo, are either registered trademarks or trademarks of The Apache Software Foundation. Sponsored by Anant Corporation and Datastax, and Developed by Anant Corporation.

Get Involved with Planet Cassandra!

We believe that the power of the Planet Cassandra community lies in the contributions of its members. Do you have content, articles, videos, or use cases you want to share with the world?