Docs

    AWS Kinesis Throttling

    Each Kinesis shard has 1 MiB of data per second or 1000 records in write capacity and 5 read transactions in read capacity. Attempting to exceed these limits results in throttling and your producers and consumers will not be able to carry out their operations.

    Causes of throttling include:

    • Hot shards: throttles are caused by a few shards in the stream that receive more requests than the average shard. This is usually caused by uneven partition key distribution
    • Not enough capacity: throttles are caused by not having enough shards to service the number of requests your application requires

    How Monitoring Works

    Streams have two dimensions: stream level and shard level. While stream-level metrics offer an overall view, shard-level metrics provide detailed insights specific to individual shards.

    Relying solely on stream-level metrics may lead to inaccuracies, especially in scenarios where data distribution across shards is uneven. By focusing only on shard level metrics, Blue Matador can pinpoint specific issues within individual shards, such as capacity constraints or uneven data distribution. 

    This granular level of insight enables proactive identification and resolution of potential bottlenecks, ultimately optimizing the performance and efficiency of Kinesis Streams data processing workflows.

    • Read Metrics
      • For monitoring data ingestion, Blue Matador tracks the IncomingRecords and IncomingBytes metrics. These metrics provide insights into the volume of data being processed by the stream on the read side.
    • Write Metrics
      • On the write side, Blue Matador monitors OutgoingBytes. This metric helps track the volume of data being written to the stream, providing valuable information about outgoing data traffic inside the shard.

     

    Hot Shards

    Kinesis assigns your records to a shard by taking the MD5 of your partition key. To find a hot shard, you can log the MD5 sums of your partition keys and then check them against the range each shard returns through the describeStream API. If you find that a disproportionate number of partition keys fall into a range, that shard is a hot shard and should be split. Conversely, shards that have disproportionately less records can be merged to consolidate the capacity needed for those records.

     

    Not Enough Capacity

    When your application grows, you will send more data to your Kinesis stream and eventually need more capacity. In this case, you will have to reshard your Kinesis stream to add more shards. Most of the time, a separate application will monitor Kinesis and automatically reshard as necessary. In fact, AWS provides an open source application for this purpose.

     

    Resources