Data integration, System integration, Extract, transform, load, Business intelligence, Information technology Apidays Live Paris | Innovation and rejuvenation combined: a beneficial approach mixing API and ESB
This event is a lot about api. This epa days right and what we see in what we do at our customers, is that we still have to combine this new generation approach of api with the old generation approach of esb and i’m, going to explain what can be the benefits of using both, because Sometimes you can think okay there’s some overlap, and we don’t really want to do that, but actually from a system architecture, point of view. That might be interesting. First, just a quick word about astrakhan, which is uh. The company i created in 2012. we’ve been publishing some a lot of information about what we do with when we create this data and digital integration systems and actually that’s uh. This conference is about our return on experience. So, to illustrate what i will present today, i will use a return on experience that we have with uh one of our customers uh in france um, who asked us last year to design part of this information system, which is at the middle of the the information System which we call the the middleware to help them actually achieve several goals and stakes at the same time, the first thing was that they were working. Part of their information system was not in the cloud, but it was uh hosted by a partner, and this partner was let’s, say quite of a old school, not that reactive. So there were a lot of latency each time they had new needs and new requirements.
So they wanted to migrate a part of their information system back from the hosting partner to their internal information system. So they had to deal with the logic of migration of the information system, moving back a part of their assets, uh from this hosting partner to their own in integration system. But actually, as you know, uh migration projects, they are not very easy to legitimate um. You need you most of the time new use cases to get the budget to trigger this kind of initiatives so um. They also worked on exposing apis to some other external partners. Business partners, in that case, in order to extend their activity, create new business models and finance in a way, this migration part this migration sub project by creating something new for for them existing information system. So this project was actually quite tricky because we had to deal with uh data flows, moving from the hosting partner and duplicated with the new uh, the new architecture of the information system file transfers because we had to deal with file transfer. It was not only api and service based architecture; it was everything at once, so we needed to create something that could be relevant in that context and we call that a hybrid integration platform, because it was part on premise and partly in the in the cloud. So that’s what we are going to see so first, this notion of hybrid integration system platform is not something that comes out of people at astrakhan it’s, not something that is exclusive to what we do.
Actually, this is something that gartner talks a lot about um and which makes a lot of sense uh when you are on the roadmap of digitizing, your um, your business uh, you can see on this slide actually, on the right hand, side you see the api part Which is uh creating new ecosystems uh having opening your information system, and you can see in in gray on the right uh bottom part: the information system platforms, the backend, the existing systems that you still have to deal with, and some of these backends. Sometimes they are not accessible through apis because they are actually symbolizing the the old generation of systems, so that’s, usually the kind of things we have to deal with when we do uh this system architecture, uh issues it’s, not software, uh architecture. We we don’t have the full freedom to design the the system. The way we would like to we have to deal with all these old generation cohabitating at the same time, and we have to find connectors and we sometimes we find esb already in place and and we need to integrate esb with the api, because we just can’t Get rid of the of the esb and and that’s the point of this of this talk so, of course, it’s all about creating a mediation, uh layer that will uh assess all the problems we can have when we try to interface systems today, this mediation layer is Allows a lot of features that we can see on the right hand, side of this page authentication authorization things that can be provided by an api management platform, things that are sometimes provided already by an esb and that’s how we start to see what an hybrid integration Platform is actually we most of the time we have to deal and combine the features and the advantages of such platforms as uh.
Api management, uh enterprise service bus etl, extract, transform load, sometimes manage file transfer for sure. When there are some, we need to combine all these platforms together to create what we call actually the transverse information system. The information system which is in between that will be extended thanks to uh the the help of the api management platforms, but which will make even more sense if we can create something that combines use cases combines formats from data to file to apis and back to Files sometimes – and this is uh allowed by the fact that most of the time when we start working on this kind of architectures, the customer, they already have actually some middleware layers working. So what they need now is to rationalize this middlewares layer and they need to create some kind of global governance and global doesn’t mean centralized. We don’t want to centralize things. We want to be able to make a benefit out of the microservices architecture. We want to use the containers it’s possible today we are going to see uh to see how, but, logically they want to create this kind of layer of mediation that will allow him to allow them to to govern their whole integration strategy and today, integration strategies in Large corporation is made with files is made with data is made with let’s, say all generation services and made with new generation apis, and we need to create a global governance for all of this.
So it means that this mediation layer, this mediation layer that will talk about here uh, can be uh, centralized on on several components: uh, including api gateways, sometimes uh collaborating with uh esbs when they are here. Sometimes they are there. There are no usb, but it can make sense to have a combined approach of api and esp gateways together in api gateways and esb together and that’s what we are going to uh to see uh now uh, we see now current deployment of containerized esb uh i’m. Now, working in asia and that’s things, we do i’m working for a company which has a strong api development policy, but still they keep on using esb for different kind of use cases and they have containerized this esb because of course they have some stakes on distribution And deployment of services that cannot be assessed with um tackled with a central esb but uh. Actually it works fine and i would see no other way to uh to to build that kind of architecture than making api management and esb collaborate in that case. So we are actually building complex integration systems. We are building complex mediation layers that are making all these technologies uh discussing with each other sharing the features. Of course, api can be fully distributed. Esb, as i said, can be today fully distributed, but mft most of the time they are technologies which are still monolithic, so we cannot all the time be.
We are not able to design those systems the way we would like to that’s as simple as that, but the integration world is made of technologies, cohabitating uh with each other, and we have no choice than doing this. Actually, sometimes, we also extend the capabilities of such an integration platform as such a mediation layer by blending it with master data management, data layers, business process management, all the things that extend the business capability of such a platform in order to make it smarter from a Business point of view and create uh, also new services at the information system level. This is all uh connected with the fact that uh those integration technologies they haven’t been replacing uh each other um. Over time we started in the past doing files. Then we made data with etl, then we made services with esb. Now we do api management, but none of these technologies have actually disappeared. They are still here and actually as long as they are still here. The best way to work with them is to provide some kind of um layer, which makes sense from a from a logical point of view, of course, behind that we generally have several kind of technologies working together. They are not most of the time provided by a unique vendor. Some of them are open source. Some of them are not, but the idea is to provide a global layer, as i said, of governance, of administration and of monitoring, because when we create this kind of system communication system, the most important thing to do is to have a command monitoring to monitor.
All the flows and all the interactions with api with files and so on, and usually we do that these days with a combination of elasticsearch and kibana, because it’s uh the the easiest way to proceed. Uh, when we see all the kind of formats we have to deal with that’s, just an example taken from the the project i presented you at the beginning of the speech. You see that list of uh flows that application needs to uh need to exchange with each other, and you can briefly see on this slide that we have several kind of protocols. We have several kind of formats. We have uh some um flows that implied the on premise: part of the information system. Some flows, uh are making work the hosting partner, and some of them are completely located in the cloud because all the uh, the mediation layer itself, has been uh deployed in the cloud with the ibm, managed services actually that’s. What you see here so that’s the architecture of the the project we we described of course, it’s a simplified, uh presentation of such an architecture. And what do we see here? We see the integration services based on appconnect enterprise, which is actually Music. How do you say ibm’s esb? We see here the silo of data, meaning the data which are transferred from the hosting partner, on the left hand, side to the central uh data repository over time slowly over time, transactional data.
They are pushed here by data flows. They are pushed there by files. Sometimes we do a little bit of service, but most of the time, these are some kind of flows that each time a customer is created each time a customer is updated. We update this repository and we make it a full repository. A full 360 degree customer view over time. We do that by integrating this the application using the connectors that are provided by the the ibm esb, and then we expose all of this intelligence uh with the api connect platform, which is the api solution that ibm provides and, of course, we make some. There is some interest of using ibm on both sides of uh, this uh, this architecture, because the components actually they are uh consolidated together and they work fine together. So they are sharing the same, actually, the same fabric of of technology, a little bit more uh detailed architecture, it’s, basically quite the same. What we see here is a little bit. Uh is the description of some of the flows that we can uh, that that we can see there and uh, and these flows actually uh. They are. They have been uh codified in order to make the full system completely uh maintainable and when we do the monitoring of the flows, because, as you can see, the data is actually moving all the time in the information system. We have this kind of codification that allows us to translate the a5 or c1 let’s say into customer flow.
Customer data flow, of course, we’ve been doing modeling of the data. This full system uses canonical formats in order to have one single format at the the scale of the information system, and actually that allows us to uh not to connect not to link all the application and all the mediation layer together. But to keep a little bit of loose connection between the application and then to deploy our migration uh our migration policy, because that’s the way we do things we don’t want to connect application together, especially when there are new applications coming up and all the applications that We want to decommission in the in the midterm a few words now about how the project was done in agile. I just wanted to say here that when you do when we do api projects, when we do middleware projects more generally, when we combine architecture and development of orchestrations and flow it’s possible to do it in agile, sometimes we don’t think we should do that because it’s, Not that business oriented, but if you think a bit about this the flows themselves, they are manipulating uh business data, so they’ve got business value and it’s actually very valuable to create all these uh flows over time during sprints, because you can actually see you can actually See what we can reuse from an architectural point of view throughout the building of such a construction, something you should be care about, especially when you are working with the managed services.
Most of the time you your attention is uh is set on the fact that you are dealing with the volume of transaction be aware that the volume of flows is not the only way to size to size such kind of platform. They are flow oriented, but actually they are also executing in virtual process core and most of the time, the vendors they do not really pay attention about. The sizing of this course so that’s something you should be aware of. When you’re going into production, you may face bottlenecks and that’s very unpleasant to face this kind of bottleneck, so be careful because that’s, usually the last mile, and you discover that in the end and it’s, not something you you want when you are ready to go to Go live uh, one last word: uh about uh the the architecture of this kind of platforms. Uh. I wanted to conclude this uh speech, showing you, the the market segments of integration, and i uh highlighted five of them. Those five are exactly the one that uh are used when we create this kind of hybrid integration platform. So we can see today that when we want to have a global integration policy using apis, we need data integration tool. We need old school esbs, we need api. We need new ipas also, and we still need uh files, because most information systems are still working with files uh relying on files heavily. Of course, we can see the dynamics of this market when the the blue um, the blue part of the histogram, is lower than the gray part.
It means that this market is decreasing, so we can see that esb are actually moving a little bit away. So that’s the tendency and that’s what we we see in this api days, a set of conferences, and we see that api are still progressing, but we think that’s in the middle term. In the long term, even we will still have to combine all these technologies together.