How did I create SAI
CREATE INDEX sai__my_table__my_column ON MY_KEYSPACE.my_table(my_column) USING 'sai';
SAI in being constructed(part of logs)
CREATE INDEX sai__my_table__my_column ON MY_KEYSPACE.my_table(my_column) USING 'sai';
Some 'Maximum memory usage reached' are logged
INFO [Native-Transport-Requests-1] 2025-05-26 09:06:29,234 ColumnFamilyStore.java:1052 - Enqueuing flush of system_schema.indexes, Reason: INTERNALLY_FORCED, Usage: 340B (0%) on-heap, 193B (0%) off-heap
INFO [PerDiskMemtableFlushWriter_0:22] 2025-05-26 09:06:29,252 Flushing.java:153 - Writing Memtable-indexes@327963785(171B serialized bytes, 1 ops, 340B (0%) on-heap, 193B (0%) off-heap), flushed range = [min(-9223372036854775808), max(9223372036854775807))
INFO [PerDiskMemtableFlushWriter_0:22] 2025-05-26 09:06:29,253 Flushing.java:179 - Completed flushing /bitnami/cassandra/data/data/system_schema/indexes-0feb57ac311f382fba6d9024d305702f/da-3gql_0pat_1ekzk2rp94k38bttl9-bti-Data.db (139B) for commitlog position CommitLogPosition(segmentId=1748224867994, position=6623148)
INFO [Native-Transport-Requests-1] 2025-05-26 09:06:29,290 ColumnFamilyStore.java:1052 - Enqueuing flush of system_schema.tables, Reason: INTERNALLY_FORCED, Usage: 924B (0%) on-heap, 861B (0%) off-heap
INFO [PerDiskMemtableFlushWriter_0:23] 2025-05-26 09:06:29,304 Flushing.java:153 - Writing Memtable-tables@49436128(824B serialized bytes, 1 ops, 924B (0%) on-heap, 861B (0%) off-heap), flushed range = [min(-9223372036854775808), max(9223372036854775807))
INFO [PerDiskMemtableFlushWriter_0:22] 2025-05-26 09:06:29,348 Flushing.java:153 - Writing Memtable-keyspaces@491501704(154B serialized bytes, 1 ops, 340B (0%) on-heap, 171B (0%) off-heap), flushed range = [min(-9223372036854775808), max(9223372036854775807))
INFO [PerDiskMemtableFlushWriter_0:22] 2025-05-26 09:06:29,349 Flushing.java:179 - Completed flushing /bitnami/cassandra/data/data/system_schema/keyspaces-abac5682dea631c5b535b3d6cffd0fb6/da-3gql_0pat_208be2rp94k38bttl9-bti-Data.db (118B) for commitlog position CommitLogPosition(segmentId=1748224867994, position=6623148)
INFO [Native-Transport-Requests-1] 2025-05-26 09:06:29,395 SecondaryIndexManager.java:226 - Index [sai__my_table__my_column] registered and writable.
INFO [Native-Transport-Requests-1] 2025-05-26 09:06:29,396 ColumnFamilyStore.java:1052 - Enqueuing flush of my_keyspace.my_table, Reason: SCHEMA_CHANGE, Usage: 0B (0%) on-heap, 0B (0%) off-heap
INFO [SecondaryIndexManagement:1] 2025-05-26 09:06:29,407 StorageAttachedIndex.java:874 - [my_keyspace.my_table.sai__my_table__my_column] Submitting 2 parallel initial index builds over 23 total sstables...
INFO [SecondaryIndexExecutor:2] 2025-05-26 09:06:29,421 V1OnDiskFormat.java:161 - [my_keyspace.my_table.sai__my_table__my_column] Starting a compaction index build. Global segment memory usage: 0B
INFO [SecondaryIndexExecutor:1] 2025-05-26 09:06:29,421 V1OnDiskFormat.java:161 - [my_keyspace.my_table.sai__my_table__my_column] Starting a compaction index build. Global segment memory usage: 0B
INFO [SecondaryIndexExecutor:2] 2025-05-26 09:12:21,382 V1OnDiskFormat.java:161 - [my_keyspace.my_table.sai__my_table__my_column] Starting a compaction index build. Global segment memory usage: 63.323MiB
INFO [SecondaryIndexExecutor:2] 2025-05-26 09:16:17,676 NoSpamLogger.java:104 - Maximum memory usage reached (1.000GiB) for chunk-cache buffer pool, cannot allocate chunk of 8.000MiB
INFO [Messaging-EventLoop-3-9] 2025-05-26 09:16:55,623 OutboundConnection.java:1153 - cass-prod-test-cassandra-0/192.168.183.228:7000(/192.168.183.228:50412)->cass-prod-test-cassandra-0/192.168.183.228:7000-SMALL_MESSAGES-1e70a6b3 successfully connected, version = 13, framing = CRC, encryption = unencrypted
INFO [Messaging-EventLoop-3-12] 2025-05-26 09:16:55,623 InboundConnectionInitiator.java:503 - /192.168.183.228:7000(/192.168.183.228:50412)->cass-prod-test-cassandra-0/192.168.183.228:7000-SMALL_MESSAGES-2a3dbced messaging connection established, version = 13, framing = CRC, encryption = unencrypted
INFO [SecondaryIndexExecutor:2] 2025-05-26 09:31:17,676 NoSpamLogger.java:104 - Maximum memory usage reached (1.000GiB) for chunk-cache buffer pool, cannot allocate chunk of 8.000MiB
INFO [SecondaryIndexExecutor:2] 2025-05-26 09:45:13,734 V1OnDiskFormat.java:161 - [my_keyspace.my_table.sai__my_table__my_column] Starting a compaction index build. Global segment memory usage: 396.754MiB
AFTER 3 HOURS
From the logs, I thought the SAI is created within 3 hours.
INFO [SecondaryIndexManagement:1] 2025-05-26 12:14:04,544 SecondaryIndexManager.java:1860 - Index [sai__my_table__my_column] became queryable after successful build.
However, the table is compacted again after 12 HOURS
This is time when I am already able to query the table with SAI
INFO [CompactionExecutor:660] 2025-05-27 01:09:23,044 CompactionTask.java:167 - Compacting (39aa7330-3a97-11f0-b61d-d569d536760d) [/bitnami/cassandra/data/data/my_keyspace/my_table-5113370028ff11ef910297a7d29eadf0/nb-29945-big-Data.db:level=0, ]
INFO [CompactionExecutor:660] 2025-05-27 01:09:23,060 V1OnDiskFormat.java:161 - [my_keyspace.my_table.sai__my_table__my_column] Starting a compaction index build. Global segment memory usage: 0B
INFO [NonPeriodicTasks:1] 2025-05-27 01:14:03,135 BigFormat.java:231 - Deleting sstable: /bitnami/cassandra/data/data/my_keyspace/my_table-5113370028ff11ef910297a7d29eadf0/nb-29945-big
My questions
- How to handle logs 'Maximum memory usage reached'? What metric from the cassandra.yaml should I increase, the file_cache_size ? If I increased the metric, will it be a problem or bring extra burden to the database?
- The compaction seems promising, but why there are logs about the compaction and new SAI again in 12 hours, when the SAI is already queryable and writeable?
- Any suggestion to shorten the SAI creation process with some workaround?