Migrating to Amazon API Gateway: A Datalex Success Story

Written by: Ronan Prenty, Solutions Architect, AWS in collaboration with Daniel Morrow, Engineering Director, Datalex, Alberto Galan, Infrastructure Engineer, Datalex and Jeff Stivers, Head of Knowledge Management, Datalex

In 2020, Datalex had an impending renewal of a multi-year deal for their APIs with their existing provider. As part of a best practice review, Datalex reviewed other options. After bringing this to the attention of the AWS Account Team, a demo and thorough evaluation of Amazon API Gateway was shown to Datalex. Datalex was impressed by its rich integrations, flexible pricing model and compatibility with OpenAPI Specification (OAS). Datalex APIs were consumed by applications out of Datalex's direct control. This meant that any API migration needed to be totally invisible and have no impact on existing customers.

This blog post explains how Datalex migrated REST APIs from another provider across all environments to AWS within two weeks.

Considerations

The initial migration of Datalex REST APIs to Amazon API Gateway needed to be accomplished quickly, without any impact to Datalex's existing customers and be totally invisible. In the API platform that was being replaced, applications consuming Datalex APIs were generally out of Datalex control. Speed, maintaining service to customers and ensuring zero hands-on work were critical to the success of the migration project.

The two primary considerations were to import existing API keys and maintain the existing structure of HTTP requests.

Importing existing API keys

Datalex customers were using pre-existing API keys in production environments. If a change to the API keys was required, the goal of a seamless migration could not be realized. Datalex used Amazon API Gateway APIs to import existing API keys to complete a seamless cutover and avoid the need for new API keys. Amazon API Gateway usage plans allowed Datalex to improve control by implementing quotas and throttles at the API key level, enabling capabilities that Datalex did not have with the previous API platform.

HTTP request structure

The existing structure of the HTTP request was to send the API key to the former provider in a format that was not supported by Amazon API Gateway.

Facilitating the invisible swap

To facilitate an invisible migration whilst maintaining the HTTP request structure, Datalex used AWS Lambda custom authorizers to ensure calling clients were not impacted in the process. This made it possible to intercept the request and retrieve the API key without having to change the HTTP request structure.

API Gateway has two options for the source of the API key: HEADER or AUTHORIZER. Initially, Datalex used AUTHORIZER as the source for the API key. The Datalex migration team was able to obtain the value of the API key in the Lambda custom authorizer code and send it back as usageIdentifierKey in the response payload. API Gateway uses this value to validate the API key against a usage plan to decide whether a request will be allowed to the backend.

After ensuring that all HTTP methods and resources were represented in the APIs Datalex created in API Gateway and required an API key, Datalex executed existing tests against the newly built APIs in development. This proved everything worked as expected before Datalex moved to production.

After validating success in dev/test environments, the migration team made a CNAME change to point Datalex's domain name to API Gateway instead of the previous provider. With that, Datalex achieved their goal of a zero-downtime migration that was invisible to customers and end users. All Datalex customer API environments were now on AWS.

Thinking long term

The partnership between the Datalex API team and the AWS team was productive and the AWS team provided consultative services throughout the project. In the existing API setup, Datalex was accepting requests to REST APIs from clients with the API key in a format not supported by API Gateway.

After a careful review with the AWS team, Datalex moved to a structure where the API key is sent in a natively supported method, the x-api-key header. Datalex was able to reduce costs by eliminating Lambda authorizer invocations to facilitate API key mapping.

After the successful invisible migration to API Gateway, Datalex sent depreciation notices to their clients, to get them migrated to the new HTTP request structure. In the words of Daniel Morrow:

'By representing our infrastructure as code, this allowed us to quickly clone the API and deploy a parallel version which sources the API key from the x-api-key header. We gave our customers a timeframe in which they needed to point their clients to the new API endpoint which takes the API key in a new format. During this time, both sources were supported. Once all customers were leveraging the new endpoint, the AWS CloudFormation stack representing the original API was deleted.'

Instead of invoking a Lambda function when a request arrives, Datalex can now fully rely on API Gateway usage plans for fine-grained quotas and throttling.

Business outcomes

Infrastructure as code
By using AWS CloudFormation, Datalex was able to develop a repeatable, baseline API architecture represented as YAML code. Developers can also use version control when they make changes to APIs. API Gateway and AWS CloudFormation have removed the heavy lifting of the previous process so developers can innovate at speed.

Fine-grained quota control
By using AWS usage plans and API keys in API Gateway, Datalex has more control over individual API keys. This allows Datalex to allocate a certain number of requests per month on a per-customer basis. It also enhances Datalex's capability to monetise future APIs.

Cost savings
By migrating to API Gateway, the cost to operate the API solution has dropped by an order of magnitude from the previous implementation. Compared to the pricing agreement with their previous vendor, Datalex has saved a significant amount across all of their REST APIs.

Payment model
Instead of an up-front fixed payment, Datalex wanted a more flexible payment plan based on actual usage instead of projected committed usage. The AWS pay-as-you go pricing model better suits the Datalex business plan and is better tied to Datalex's transaction-based pricing model.

Tight integration with AWS services
Through the partnership with AWS and use of API Gateway, Datalex realised benefits that include a rich set of integrations with AWS Lambda, HTTP endpoints and direct integrations with Amazon Simple Queue Service (Amazon SQS), Amazon Simple Notification Service (Amazon SNS), and Amazon Kinesis. For information, read the Datalex and AWS Are Modernizing Airline Retailingblog post.

Future migrations
After more Datalex applications are migrated from the on-premises environment to AWS, API Gateway will facilitate a seamless, straightforward integration between Datalex applications.

ConclusionDatalex was looking for a more cost-effective solution that would allow the company to scale as its business grows and use cases are added for their customers. The AWS native integrations in API Gateway, such as Lambda custom authorizers, allowed Datalex to make a clean, invisible migration without impacting customers and their end users. The AWS solution makes it possible for Datalex to integrate directly with AWS services and represent the Datalex API footprint as code in a pay-as-you-go model.

In the words of Daniel Morrow, the Engineering Director at Datalex, 'We completed a seamless cutover from our previous provider to API Gateway. We are now fully cut over on all production and non-production systems and no longer have any use for our old provider in Datalex environments. The success of this project will have a big influence on our use of API Gateway when it comes to migration time from the data centre.'