AWS Kinesis Iterator Age

The data retention period in your Kinesis stream dictates how long data will be preserved in the stream before it is read. If data is in your stream for longer than the retention period, it will be lost. By using the GetRecords.IteratorAgeMilliseconds CloudWatch metric to approximate the age of data in your stream, Blue Matador will automatically warn you if your data is going to expire.

Avoiding Data Loss

If your iterator age is at half your data retention, you should consider taking action to ensure your data does not expire.

One solution is to increase the number of consumers for your stream so that data is processed more quickly. Depending on how saturated your shards are, you may need to split shards in your stream to allow for increased throughput.

If your stream usage is bursty, then it may be the case that your current consumers will be able to catch up on processing when usage is lower. If this is the case, then make sure the data retention you have configured is appropriate for your workload. You may consider increasing retention to ensure no data is lost.

In any case, if you expect your Kinesis data to be consumed nearly-instantaneously, then you can optimize application code and add consumers to reduce the delay in processing as measured by the GetRecords.IteratorAgeMilliseconds metric.

Resources

• Monitoring the Amazon Kinesis Data Streams Service with Amazon CloudWatch (AWS Documentation)
• Changing the Data Retention Period (AWS Documentation)
• Kinesis Data Streams Limits (AWS Documentation)
• Splitting a Shard (AWS Documentation)