"What are the most significant changes to the integration of applications and data?"

• Focusing on APIs the old way will not work in the future. We need to search for new ways of doing integration moving away from heavy middleware to mobile lightweight native APIs. This requires a different architecture strategy.
• Proliferation of publically consumable cloud services with REST and JSON enabling you to quickly bring services to the market and therefore value to customers more quickly. Microsoft is embracing the movement and beyond. Slack has had tremendous growth with an open standards based API.
• The needs and requirements change at the same rate as big data but the solutions don’t come along as quickly. Enterprise manufacturing and pharmaceuticals have a significant need for pushing more data through their system faster. They’ve hired chief analytics officers but they still aren’t able to process data in real time because the average time to build an integration is six to 12 months while the need for real-time information is growing stronger and companies need to be able to pivot based on new information on a daily basis. We need to build platforms that allow customers to update and change integrations at will.
• Microservices have been introduced. We’ve gone from virtual machines to microservices. Know which data goes where and why. 100 times more complex. A containerization platform can handle volumes of stateful and stateless data.
• Reduction in the cost of storage as a result of the cloud via Redshift and S3. You can get a Seagate 10TB hard drive for less than $600. Unlimited storage at your fingertips for a very low price.
• The fact that 74% of organizations are now involving business people in IT due to the demand for innovation, speed to market, and real-time data ingestion and analytics.
• Multiple platforms – mobile, IoT, and cloud – have fundamentally transformed integration. 1) Look at what is being integrated. 85% of our clients are hybrid – ESB and EPL are no longer pertinent. 2) Who does the integration? Digital connections can bottleneck with a small group of tech developers. It needs to be decentralized across the company and be handled by business analysts who understand business processes and data. For a technology to go mainstream, it must be accessible by and to people who understand business needs. 3) Where does integration live? No longer ESB, SOA, or on-prem. Clients want to go cloud-first with integration moved to the cloud. The smallest possible footprint is behind the firewall. Network across partners, providers, and customers all have the inherent need to connect.
• 1) Industry is moving from SOAP to RESTful API. 2) Architecture applications are beginning to build and distribute systems to scale and handle workloads. Movement towards RESTful APIs versus traditional SOAP. Build distributed systems to handle workload and scale appropriately.
• 1) Blending of batch integration with real-time and streaming data. Cycles are compressed. Data is now at different levels of latency and speed. 2) Data drift – data has evolved from a semantic structure that was predictable. Now it happens with more frequency due to data access going from transactional with clear APIs, like SAP and Oracle interfaces. Today there’s a lot more event and interaction data that is streaming. Systems don’t expect downstream consumers to access the data in the form that it’s in.
• The amount of unstructured data doubles every 18 months with IoT data and the propensity to hold data in perpetuity. Statutory requirements go away. There’s a challenge in managing and getting value to create an asset versus a cost. Nobody wants to delete data due to the fear factor of being the person who decides to delete the data.
• Additional knowledge and functional requests. We used to look for the minimum viable product versus how to scale. Security is a never thought. Scale of integration is massive with the number of connections and platforms with IoT, mobile integrating requirements. REST APIs have exploded the last five years. People want to know how to be more efficient and responsive. No one is tackling reactive for mobile. Ability to get a reactive responsive layer without having to make the Capex expenditure is appealing. Driving big data real time – dynamic, mobile, IoT is event driven data. People want to use Reactive, Ember, and Angular. There’s a push towards microservices as we become a more event-driven world. API doesn’t play naturally to event-driven data. Moving data in real time must be monitored and able to do it quickly. People are trying to do integration who don’t come from a mobility of IoT background. How do we make it easy for people to use? Gaming is huge with real-time data coming in, for betting, and in-play betting. You can enable customers to presubscribe to topic or dynamically force subscription.
• APIs are driving speed. Cloud and SaaS are driving integration beyond four walls. Innovation around digital and triangulating data, people, and processes in new ways. Developers developing apps is now an accepted practice and requires creativity on the part of the developers with a focus on UX, without bottlenecks accessing data, security, and availability at pace. We’ve gone from compartments to EAI, SOA and now an API foundation for digital integration. There are outside ecosystems. APIs are now available as a channel.
• API middleware is replacing SOAP with microservices and RESTful APIs. There’s a proliferation of applications, databases, and interfaces – Hadoop, Spark, Cassandra. RESTful APIs and web hooks turn into URLs analog to RESTful API that can turn into web calls. We see the web hooks side growing since users can call and say they have something for them.
• The most significant change to integration has been the growth of cloud adoption. Now more than ever, a hybrid integration platform is key to surviving in the digital age. Anypoint Platform enables organizations to seamlessly connect applications, data, and devices, both on-premises and in the cloud.
• It's hard to create test environments for integration in a timely fashion. It's hard to sanitize data for secure integration testing. Reliance on IT groups such as DBAs, storage admins, and other staff to turn around requests that are integration team is dependent. Volume of data ever increasing, particularly in heavily regulated industries like finance and healthcare. Increase spread in applications across more and more sites, include cloud. Increasing number of components that have to be integrated. It’s not two to three applications, it’s now dozens and more are on the way. Increased pace of change for each application in an organized suite. Increase in the types of data repositories that must be supported (big data, file system data, RDBMS...)