AWS re:Invent: Why modern applications are becoming unthinkable without integration?
Gregor Hohpe is a Director of Enterprise Strategy at AWS and author of several books for software architecture, cloud strategy and integration patterns. Before joining AWS, Gregor has worked with the Singapore government, as Technical Director in Google, and as Chief Architect at Allianz SE. But maybe the most amazing about him is his skill to explain things in a super simple way.
His lecture at AWS re:Invent “Building modern cloud applications? Think integration” shows in a very simple and straight forward way that modern applications will soon be unthinkable without integration.
“Integration becomes an integral part of building applications. You’re deploying integration at the same pace you’re deploying software, and you’re using the same methods. So we do not call it integration anymore, we call it modern applications.”
Gregor Hohpe Director of Enterprise Strategy at AWS
The topic of integrations seems to be new and relatively simple, but it has actually been developed in the last few decades. We’ve been through different integration approaches, like migration, ETL, standard data synchronization, enterprise service bus, and recently serverless event-driven architecture.
If we try to imagine the simplest integration possible, it would be connecting system A to system B. If we have 4 systems that are connected in this way:
Then they will be dependent on each other — if C fails, B will no longer be able to connect to D.
If we connect them in another way though:
They will have different dependencies and even if we remove C, B will still be able to connect to D.
How your components are interconnected defines your system’s essential properties. In the above example, we see that the lines (connections) are as important as the components. The lines represent an integration — how the systems are connected. The lines are a part of what we call Software Architecture.
Each structure comprises of software elements, relations between them and properties of both elements and relations. We cannot build modern systems without lines.
“Great architects are like great chefs. It’s not just about selecting ingredients; it’s how you put them together.”
Gregor Hohpe Director of Enterprise Strategy at AWS
What are the newest trends in system integration?
Let’s explore three relatively new terms that are becoming more and more important in the world of integrations and software development itself.
Service mesh
The service mesh is a way to control the lines between the components of a system. It is a dedicated infrastructure layer built into an app. The service mesh routes requests from one service to another, optimizing how all the moving parts work together. Rules for routing requests from point A to point B have always been necessary. However, what’s new about a service mesh is that it takes the service-to-service communication logic out of individual services and separates it to a new infrastructure layer. In a service mesh, requests are routed between microservices through proxies. Without a service mesh, each microservice needs to contain code for service-to-service communication. This means communication failures are harder to diagnose, and also developers are distracted between business goals and communication rules. If you have a complex microservices architecture, it can become close to impossible to locate problems without a service mesh.
Data mesh
Usually storage, processing and output of data is kept in one place, like a data lake or a data warehouse. Instead, a data mesh treats each domain of the organization as a unique customer. This way, each domain handles its own data pipelines and controls the ingestion, transformation, and integration of data. This makes data function like a product — it is uniquely designed for the needs of each consumer.
Domains are connected by a layer of universal interoperability, and it ensures specific standards are met for data governance, data security, and data observability. This is how each domain can process data without risking breaching security or operating with faulty data.
In short, the data mesh makes data available, accessible, discoverable, secure, and interoperable. It allows users to easily access and query data directly, without first transporting it to a data lake or a data warehouse. This way of quick access to data makes real-time analytics and immediate insights possible, as time to value is set to a minimum.
Serverless
Serverless is an approach to software development where developers can build and run services without managing the underlying infrastructure. The term “serverless” is a bit misleading though. The servers still exist, and the code is being deployed and executed on them. However, the responsibility for the servers’ maintenance is no longer a problem of the developer. A third-party vendor manages all the infrastructure and servers. This way, the developers are free to focus on the code and stop wasting time on the infrastructure.
Serverless is events-based. It is not necessary to have an integration server to perform data integration jobs. In serverless data integration, a function is triggered for job execution via an HTTP/SOAP request.
A massive transition to serverless will surely take time, but a significant number of organizations have already adopted it in some form. With serverless, companies can simplify dev-ops, minimize go-to-market time and build scalable, lightweight applications that easily integrate with each other.
Integrations in the cloud
“In modern cloud applications, integration is no longer an afterthought, like it used to be in the old days with ETL. Instead, integration has become an integral part of the way we design, build and operate applications”
Gregor Hohpe Director of Enterprise Strategy at AWS
How to choose the best approach? Consider the level of control that you have over the systems you are integrating and their lifecycle.
Coupling is integration’s magic word. Coupling is a measure of independent variability between connected systems. There are many dimensions of coupling. There might be dependencies by technology, location, data format, data type, interaction style, temporal or semantic dependencies, etc. To know how loosely or tightly coupled the systems are, we need to look at all these dimensions. If you have a lot of control over the integrated elements, you can have them tightly coupled and vice versa.
For example, serverless apps consist of small pieces, loosely joined. Smaller pieces have smaller blasts of failure, they scale more easily, and you could easily deploy updates. These are a lot of fantastic benefits.
Gregor Hohpe’s formula is:
Serverless + Integration + Automation = Awesome
Summary
The cloud removes many obstacles, such as slow provisioning and start-up times, high storage costs and limited transparency. Serverless solutions are freeing developers from installing, provisioning, managing and scaling infrastructure. Everything is up and running in minutes and the environment can easily scale when necessary.
More and more companies are using the combination of serverless, microservices and automation, to deploy modern applications in the cloud. This recent popularity, according to Gregor Hohpe, gave integration patterns an unexpected boost. Integration is no longer an afterthought, but an integral part of application design.
When planning to build or deploy applications in the cloud, consider integration as an indivisible part of the whole. Another option is to use a cloud integration platform that can help you integrate the apps you are already using. A no-code integration platform allows even non-technical users to set up integrations quickly. For example, ZigiOps has filtering and mapping capabilities that enable you to make complex integrations in minutes.
