Log in
Show password
Forgot password ?
Become a member for free
Sign up
Sign up
New member
Sign up for FREE
New customer
Discover our services
Dynamic quotes 


SummaryMost relevantAll NewsAnalyst Reco.Other languagesPress ReleasesOfficial PublicationsSector newsMarketScreener Strategies

Expedia : Kinesis Data Stream for Async Booking Email Handling... (Opens in new window)

10/12/2021 | 09:32am EST

Tips on various topic learned when troubleshooting issues

Record Aggregation

The record aggregation mechanism is enabled by default, which means that multiple records could be aggregated into one as stated before, to make more efficient use of available bandwidth and reduce cost.

In our architecture, in order to be resilient, we had the requirement to consume exactly one record for each read operation, so we had to write and read exactly one record each time. For this specific reason we had to disable explicitly the aggregation at KPL side otherwise we would have inconsistent behavior: KCL would have one message containing more records when we wanted exactly one.

We eventually spotted the issue and realized it was due to aggregation by inspecting the putRecords into the stream using the AWS CLI (https://docs.aws.amazon.com/streams/latest/dev/fundamental-stream.html).

Checkpoint mechanism

For each Amazon Kinesis Data Stream application, the KCL uses a unique Amazon DynamoDB table to keep track of the application's state. The KCL uses the application name, as tag, in order to tag and associate the DynamoDB table created, for this reason each application name must be unique. By default, the KCL creates the table with a provisioned throughput of 10 reads per second and 10 writes per second.

Each row in the DynamoDB table represents a shard that is being processed by your application. In addition to the shard ID, each row also includes checkpoint, checkpointSubSequenceNumber, leaseCounter, leaseKey, leaseOwner, ownerSwitchesSinceCheckpoint and parentShardId. The most relevant are the checkpoint and leaseKey which respectively give the position related to the last message consumed and the shard id of the related shard processor.

KCL checkpoint mechanism in detail:

  • at startup KCL set as default value "LATEST", if no checkpoint available in the KCL table, otherwise it will use the available checkpoint id. There is the possibility to override the default value using one of the following: AT_TIMESTAMP, TRIM_HORIZON, AT/AFTER_SEQUENCE_NUMBER or LATEST.
  • on each message consumed by KCL, you need to explicitly commit the checkpoint. It is necessary to track the last message consumed and keep the application state consistent.

Possible checkpoint values:

AT_TIMESTAMP: the estimate time of the event put into the stream. You will use this option if you want to find specific events to process based on their timestamp.

TRIM_HORIZON: the newest events in the stream, and ignore all the past events. You will use this option if you start a new application that you want to process in teal time immediately.

LATEST: the oldest events that are still in the stream shards before they are automatically trimmed (default 1 day, but can be extended up to 7 days).

AT/AFTER_SEQUENCE_NUMBER: the sequence number is usually the checkpoint that you are keeping while you are processing the events. These checkpoints are allowing you to reliably process the events, even in cases of reader failure or when you want to update its version and continue processing all the events and not lose any of them. The difference between AT/AFTER is based on the time of your checkpoint, before or after you processed the events successfully.

Resharding a Kinesi Data Stream

Resharding allows to increase or decrease the number of shards in a stream in order to adapt to changes in the rate of data flowing through the stream.
The KCL tracks the shards in the stream using an Amazon DynamoDB table. When new shards are created as a result of the resharding, the KCL discovers the new shards and populates new rows in the table.

What happen during the resharding process?

1) KPL starts to produce new messages on the new shards

2) KCL will continue to consume from the oldest shards until no more messages available and will tag those as "SHARD_ENDED". After that, KCL will start to consume from the new shards: this assures to preserve the order in which the data has arrived on the stream.

This is the only way to scale the kinesis stream throughput.

TECH NOTE: Potential side effect in case of increasing shards: given that, by design, the table provision of 10 RCU and WCU beware that couldn't be enough to handle the increased throughput. Therefore consider to increase those values properly.


Expedia Group Inc. published this content on 12 October 2021 and is solely responsible for the information contained therein. Distributed by Public, unedited and unaltered, on 12 October 2021 13:31:10 UTC.

ę Publicnow 2021
All news about EXPEDIA GROUP, INC.
11/26US Travel Stocks Hammered on New COVID-19 Variant Fears as Europe Eyes Travel Ban for C..
11/22Expedia Group to Participate in UBS Global TMT Virtual Conference
11/18ONLINE TRAVEL UPDATE : Amazon Expands India Travel Offerings; Sabre Loses Expedia's North ..
11/17Grammarly raises fresh funds at $13 bln valuation
11/15INSIDER SELL : Expedia Group
11/12S&P Affirms Expedia Group 'BBB-' Ratings, Revises Outlook to Stable From Negative on Im..
11/12PUMP / DUMP #10 : This week's gainers and losers
11/12Expedia Group and UNESCO Sustainable Travel Pledge reaches 4,200 hotels
11/09Trade TVs for trips this Black Friday with a first-of-its-kind holiday travel gift guid..
11/09Citigroup Adjusts Expedia's Price Target to $190 From $166, Maintains Neutral Rating
More news
Analyst Recommendations on EXPEDIA GROUP, INC.
More recommendations
Financials (USD)
Sales 2021 8 626 M - -
Net income 2021 -491 M - -
Net Debt 2021 3 142 M - -
P/E ratio 2021 -49,8x
Yield 2021 -
Capitalization 24 214 M 24 214 M -
EV / Sales 2021 3,17x
EV / Sales 2022 2,16x
Nbr of Employees 19 100
Free-Float 49,2%
Duration : Period :
Expedia Group, Inc. Technical Analysis Chart | MarketScreener
Full-screen chart
Technical analysis trends EXPEDIA GROUP, INC.
Short TermMid-TermLong Term
Income Statement Evolution
Mean consensus OUTPERFORM
Number of Analysts 31
Last Close Price 159,80 $
Average target price 190,58 $
Spread / Average Target 19,3%
EPS Revisions
Managers and Directors
Peter M. Kern Vice Chairman & Chief Executive Officer
Eric M. Hart Chief Financial & Strategy Officer
Barry Charles Diller Chairman
Rathi Murthy CTO & President-Expedia Services
Craig A. Jacobson Independent Director
Sector and Competitors
1st jan.Capi. (M$)
EXPEDIA GROUP, INC.20.69%24 214
TRIPADVISOR, INC.-7.05%3 696