Screen-scraping technology has been used for many years to speed the extraction and input of data by automating the user interface (UI) of an application. Consider a person entering data into an application. When performing manual data input, this person will enter data via the keyboard which transfers that information to fields in the UI of an application. If the user has millions of data-points, they might be better off writing a macro that automates the cut-and-paste of these data-points from a spreadsheet to the application. This is the essence of screen-scraping.
A common use-case for screen-scraping is to extract data from websites where no API has been provided. This is where both legal and technical problems can be encountered. A recent example of a legal issue was when European airline, Ryanair, accused Expedia of unauthorized screen-scraping of their website to access their flights and fares. Many suppliers, like Ryanair, do not want to provide their proprietary data to price comparison engines, like Expedia. Since many screen-scraping applications can be brittle, they often cause technical problems. For example, if Ryanair changes their website, then Expedia’s screen-scraping capabilities will likely break, making it poorly suited to mission critical applications.
Despite these challenges, screen-scraping has been given a new lease on life in the form of robotic process automation (RPA). RPA proponents claim that the marriage of AI and traditional screen-scraping techniques provides customers with increased value. An IBM spokesman has been quoted, saying “As machines are quickly learning to complete the repetitive and time-consuming tasks that take up much of our workdays, workers are being freed up to think more creatively and ambitiously about their jobs.”
From a process perspective, RPA is focused on speeding manual data-entry. Let’s use the example of a call center where agents need to enter data into multiple systems (aka swivel-chair integration). If call centers can cut a small amount of time from this process, which agents execute many times a day, there are potentially big savings to be had for the entire call center.
Now close your eyes and visualize fingernails running down a chalkboard. That horrible feeling is how integration developers generally feel about screen-scraping. Many think that a fit-for-purpose API serves as a better solution than building a UI macro (aka a robot). However, with the tools available today, developing an API is harder than assembling a UI robot. The call center owner in the example above would likely not have access to the skills to build the right API. They would need to call IT, who likely is working on other high priority projects. Alternatively, to get an RPA robot cranked out is a snap! All they need is an intern with a PC. For this reason, RPA is typically sold as a non-invasive way to integrate applications to the process owner, and not to core IT or developers.
Closer inspection here illuminates the strip between application and integration. The application is owned by the process owner and business stakeholders, while integration is owned by IT and developers. I have traditionally viewed the strip that separates the two as the chasm — to be crossed at one’s peril! One clearly knows on which side of the chasm they lie. As a Muley I have my feet very firmly planted on the integration side. As I look over to the other side, I see my Salesforce colleagues building vertical applications. In between we find a assortment of bridging technologies — part application, part integration. These include RPA, business process management (BPM), low-code, no-code, webForms, and optical character recognition (OCR). It’s inaccurate to place these technologies into a single bucket, but for the purposes of discussion, let’s call them application-integration solutions.
Application-integration solutions are increasingly being leveraged as customers demand the ability to rapidly assemble customized experiences from pre-packaged back-end services. If I can expose many of my ERP, CRM, and partner capabilities in the form of APIs then I can rapidly assemble experiences that are much more tailored and differentiated than out-of-the-box applications could ever be.
As integration professionals how can we help facilitate this?
- Standardize on APIs – Increasingly integration patterns are normalizing around API-based integration. APIs are a common framework to allow application-integration solutions and applications to bind to the integration/services layer. This simplification and standardization allows the application developer to choose the best implementation for the job at hand — packaged application, custom application, application-integration, or headless.
- Make it easier to create APIs – In many cases, an off-the-shelf API is not fit for purpose. In such cases the application developer will need to craft an API which enriches other APIs. Technologies like FlowDesigner assist with this process. Firstly, by creating a new API we keep integration logic out of the application layer reducing application customization. Secondly, we expand the market for folks who can build an API. Building an API should be as simple as pulling data from a file into Excel. The net effect of this will be to reduce the dependence on application-integration solutions and core IT.
- Target application developers with SDKs – To simplify API consumption we can provide calling code in the developers native language, along with sample code and data. There are many tools out there for provision of such code (see: https://raml.org/developers/build-your-api)
- Partner with application and application-integration vendors – MuleSoft has worked to build a strong ecosystem of application and application-integration vendors. We provide connectivity to leading BPM vendors including Pega and Appian. In order to accelerate connectivity we have provided pre-packaged integrations (aka accelerators) to many of the common applications on the market.
- Be neutral – In all modern enterprises we accept that there will be a diverse portfolio of application and integration solutions deployed. This portfolio will change over time as businesses morph to support the demands of their customers. In this environment it is critical that your integrations remain flexible to support all integration and application patterns.
Finally, as an integration developer, I should relax my prejudice toward application-integration solutions. This helps to broaden my aperture of what integration is, and helps me to position technologies, such as RPA, in my overall application network. I may not like the fact that my line-of-business managers are achieving system integration through screen-scraping, in the same way that my application owners don’t like low-code solutions built to support mobile use cases. I can now look at the overall portfolio and see RPA as a tool that can help achieve some short-term automation wins that serve to inform my longer term API architecture and integration roadmap.
For more on how to automate business processes, download our whitepaper.
