Engineering Blog

Cayan's Agile API Versioning Process

One of the things that Ken Schwaber and Jeff Sutherland recognized back in the late 1990s is that software development is an iterative and always unfinished process. The targets you’re trying to hit are moving, and to be successful, your organization must continually evolve. In their Agile Manifesto, Schwaber and Sutherland emphasize the importance of responding to change and ongoing collaboration with their customers.

Since then, countless companies have adopted some flavor of Agile as their software development methodology, and Agile strategies are now the dominant SDLC in the industry. Products are designed and delivered in short, regular iterations and are constantly evolving.

One of the more recent trends in the software industry is to treat the APIs we develop as products. The “API as a Product” movement recognizes that companies have wholeheartedly embraced APIs as a way to expose business capabilities to both external and internal developers. APIs promise the ability to experiment quickly with new business ideas by recombining core capabilities. Teams that build APIs should understand the needs of their customers and make the product compelling to them. APIs that are Products are also improved, maintained and supported over the long term.

APIs – specifically SOAP APIs - are principally how Cayan delivers value to its ISVs. Like all of Cayan’s products, our APIs are constantly evolving and adapting to better serve our partners and our merchants. Whether it’s a small enhancement or breaking out into new verticals, our APIs must evolve to deliver incremental value to our customers.

As you might expect, there are different ways to evolve an API, each with their own pros and cons. Microsoft has written a good summary of the API versioning approaches one might take:
  • Agile Versioning: Rely on backward compatibility for as long as possible and avoid formal contract and endpoint versioning until compatibility is broken. This approach is useful in agile environments that require frequent updates to production code.
  • Strict Versioning: Perform formal contract and endpoint versioning for any change to a service contract, data contract, message contract or other contract-related or endpoint-related changes. This approach is best in environments what have less frequent production updates or that require detailed tracking of any and all changes.
  • Semi-Strict Versioning: Perform formal contract and endpoint versioning when contracts or endpoints are modified in a way that your policy requires tracking the change. For example, your policy might be to allow new operations to be added to a service contract, but if any changes are made to existing operations, or if any data contracts change such that they are semantically different, it requires versioning. This approach lies somewhere between agile and strict versioning.
An agile approach to versioning – illustrated below – means making changes to existing data contracts and service contracts without versioning them, or supplying new endpoints. Strict versioning means providing new data contracts, service contracts and endpoints.

Versioning Illustration

As you might’ve guessed, Cayan’s APIs follow an agile approach to versioning. This makes it easy for us to rapidly deliver new features to our customers. In doing so, we promise to never break backwards compatibility with our clients. This means that:
  • We will return new elements in our APIs’ response messages. Clients are not required to understand or act upon these elements. Clients must not fail to parse or accept these messages when they contain “unexpected” data. Clients must, at a minimum, ignore these new elements, and fail gracefully. Clients must not perform strict schema validation against a response message.
  • We will add new methods and behaviors to our APIs. Clients must behave as per the above.
  • We will not add new required parameters to our APIs’ request messages. Any new parameters we add to our APIs’ inputs will be optional, and have sensible default values, preserving “backwards compatible” behavior for our clients whenever possible. Should we need to add a non-optional parameter or change behavior in a way that does not preserve backwards combability, we will follow strict API versioning, and introduce a new API endpoint.
Our data contracts are mutable in a behavior-preserving way, and we require our API clients to behave accordingly - Postel’s law / Robustness principle in action. Failure to do so can be catastrophic. A fragile API client will break when we introduce changes, leaving merchants unable to process credit card payments. Fragile API clients deployed in the field might cause Cayan to need to roll back a release, and corrupt our whole ecosystem. Or worse - it would prevent us from being able to release changes to our products.

Because we have an agile versioning strategy - but we do not have control over our APIs’ clients - we’ve taken several steps to ensure that our ISVs will not break as we evolve our APIs. That they’re honoring our agile versioning strategy by not performing strict schema validation.
  1. We’ve included tests in our certification process that ensure that our ISVs can handle additive data in our response messages.
  2. We’ve implemented an extension point class in several of our APIs that helps us ensure – into perpetuity - that API changes won’t break our clients. By always returning data elements in our responses that don’t strictly validate against the WSDL contract, we force our API clients to be lax in what data they receive and how they handle our APIs’ responses.
We’ve published the code below for how we’ve achieved #2, in case anyone would like to take similar steps to preserve their ability to agile version their SOAP APIs.

Below is a GitHub Gist for our SoapExtensionPoint class: