Illustration Image

Cassandra Java driver: high latency when executing many async reads in a loop

We're using DSE 6.0.18 (Cassandra version 3.11) and our application is both read and write heavy.

I have a situation where I need to fire N number of read queries for each API (by partition key) using session.executeAsync(...) in a loop. The number of queries (N) differs for each API based on the use-case. Most of the API's will fire less than 30 queries and some 20% of API's fire large no of queries like up to 50 to 300. For the APIs with a lesser number of queries fired, the overall Cassandra response time is less than 1 to 1.5 secs at max, but for APIs with a larger number of queries, the response time drastically increase to 20 to 30 secs.

I'm executing individual SELECT queries per partition key (e.g., SELECT * FROM table WHERE id = ?) using executeAsync().

My questions:

  1. Why does firing more async reads in parallel degrade performance so drastically?
  2. Is there a limit on in-flight or async requests from the Cassandra Java driver?
  3. How can I throttle or batch async reads efficiently without losing parallelism?

I read in a different post on tuning Native Transport Request(NTR), like native_transport_max_threads and max_queued_native_transport_requests. We have default values for these parameters, will tuning these help us?

Additional information:

  1. We have 12 node cluster with over 15TB data.
  2. Data is not evenly distributed and having little issues in the cluster like large partitions, uneven data distribution, older versions. We are working on moving data to newer hardware and fix all these later.
  3. No custom throttling currently implemented.
  4. On an average daily 70M to 110M reads are happening.
  5. The table has large no of cols like 400 and in the query some 300 cols were fetched each time.
Become part of our
growing community!
Welcome to Planet Cassandra, a community for Apache Cassandra®! We're a passionate and dedicated group of users, developers, and enthusiasts who are working together to make Cassandra the best it can be. Whether you're just getting started with Cassandra or you're an experienced user, there's a place for you in our community.
A dinosaur
Planet Cassandra is a service for the Apache Cassandra® user community to share with each other. From tutorials and guides, to discussions and updates, we're here to help you get the most out of Cassandra. Connect with us and become part of our growing community today.
© 2009-2023 The Apache Software Foundation under the terms of the Apache License 2.0. Apache, the Apache feather logo, Apache Cassandra, Cassandra, and the Cassandra logo, are either registered trademarks or trademarks of The Apache Software Foundation. Sponsored by Anant Corporation and Datastax, and Developed by Anant Corporation.

Get Involved with Planet Cassandra!

We believe that the power of the Planet Cassandra community lies in the contributions of its members. Do you have content, articles, videos, or use cases you want to share with the world?