We're using DSE 6.0.18 (Cassandra version 3.11) and our application is both read and write heavy.
I have a situation where I need to fire N number of read queries for each API (by partition key) using session.executeAsync(...) in a loop. The number of queries (N) differs for each API based on the use-case. Most of the API's will fire less than 30 queries and some 20% of API's fire large no of queries like up to 50 to 300. For the APIs with a lesser number of queries fired, the overall Cassandra response time is less than 1 to 1.5 secs at max, but for APIs with a larger number of queries, the response time drastically increase to 20 to 30 secs.
I'm executing individual SELECT queries per partition key (e.g., SELECT * FROM table WHERE id = ?) using executeAsync().
My questions:
- Why does firing more async reads in parallel degrade performance so drastically?
- Is there a limit on in-flight or async requests from the Cassandra Java driver?
- How can I throttle or batch async reads efficiently without losing parallelism?
I read in a different post on tuning Native Transport Request(NTR), like native_transport_max_threads
and max_queued_native_transport_requests
. We have default values for these parameters, will tuning these help us?
Additional information:
- We have 12 node cluster with over 15TB data.
- Data is not evenly distributed and having little issues in the cluster like large partitions, uneven data distribution, older versions. We are working on moving data to newer hardware and fix all these later.
- No custom throttling currently implemented.
- On an average daily 70M to 110M reads are happening.
- The table has large no of cols like 400 and in the query some 300 cols were fetched each time.