Screen scrapping (sic) in the 21st century

Posted by Trevor Perry on Apr 24, 2013 10:50:00 PM


Some years ago, customers wanted graphical user interfaces from the AS/400. In March 1992, the News 3X/400 magazine printed an article titled “GUIs - The Dawn of a New Age”. In July 1995, the Midrange Systems newspaper printed an article titled “The Final Word on GUIs”. Yet, in 20i3, customers are still using green screen applications!

Certainly, there is a lot of pressure from users to provide a graphical user interface. And, most GUI projects are short, providing relief to that pressure. Often, most companies stop there, but a GUI should only be the first step in a modernization strategy. In this, the 21st century, the GUI has become commonplace among applications, yet the IBM i community drags its feet. There are several reasons, one of them being vendor noise (see more on that in an article I wrote for iPro Developer), another being that we tend to cling to a past that is familiar. The main reason, though, seems to be ignorance - a lack of knowledge and awareness of what is available for IBM i in regards to modernization.

Just this month, on a public forum, a debate raged about how green screens were fantastic for data entry. Having written/presented/talked about that particular subject for more than a decade, I should be surprised that it still exists, then I realize it fits the aforementioned main reason. Simply put, there are dozens, if not hundreds, of techniques to capture data in 20i3. A human typing data is the most error-prone and slowest technique to gather data, period. When you compare green screen keyboard data entry with browser data entry, there is definitely no comparison - green screen is faster. But that assumes you are stopping there, and not considering replacing human data entry. Once you consider the alternatives, there is no reason in the second decade of the 21st century to ever again rely on the concept of mass data entry using human input.

Furthermore, if you count the number of screens/pages where data entry is used inside your legacy application, it probably is a very small percentage. For an IT organization to ~not~ modernize based on one percent of the user application is again, based on the aforementioned main reason. Simply put, if you persist in human data entry being part of the solution provided, then you would keep that functionality inside the green screen emulation tool you are using, and modernize the rest of your application, where even the most basic GUI provides anywhere between 30 and 80 percent more user efficiency, and removes the additional cost of training a new employee in using outdate green screen technology littered with programmer abbreviations and terminology. Try asking a user to “work with your spooled files” in this century, and if they don’t leave the room, they will wonder why you are asking them to bring their knitting.

Another major wall to modernization is the term “screen scraping”. Recently, on several forums, one particular individual has been commenting with the same question to many of my own articles or posts. This behavior identifies someone as a troll, according to Wikipedia. The comments typically go something like this: “Is looksoftware using the 5250 screen scrapping to build the GUI screens?”.  It is obvious that it is the same person, due to the consistent spelling of “scrapping”. Knowing that the answer is already available from looksoftware, and in fact, from many sources, answering a troll would be an emotional reaction, and the words in my response might not be conducive to an education.

However, since this is a looksoftware blog, I can focus, and answer the question.

Screen scraping technology was used at the dawn of the GUI age by several vendors. The concept is for software on the desktop to act as a telnet client, and use the 5250 data stream in a conversation. The scrapping, er.. scraping, part comes when the GUI representation is built. By looking at the elements of the 5250 data stream, the output and input fields can be uncovered, and presented to the user in some GUI form.

Over the years, “screen scraping” has become a derogatory term in our industry, even though it was key to the evolution of GUI in the IBM i world. About five years ago, a vendor was able to (illegally) reverse engineer the Websphere handler that was added for Webfacing and HATS. Instead of reading the 5250 data stream, the conversation was now converted to/from HTML. Tapping into that HTML data stream was announced as being “modern” and no longer “scraping”. With a closer look, it was obvious that the vendor’s software was scraping the HTML to produce a result for a browser. Except, of course, most of the HTML was already generated, so the scraping algorithm was simple. All that was produced was an HTML representation of the 24x80 (or 27x132)  scraped into a browser. And that was no different than the IBM i Access for the Web product. The vendor deriding the “screen scraper” term were actually scraping screens.

In the last two to three years, other vendors have followed suit. Whether it is scraping the 24x80 into HTML their own way, leveraging an RPG Open Access handler, or writing their own translation of the 5250, the core of any good GUI tool is a translation engine. If the end result is more than a flat representation of the 24x80, then the term “screen scraping” no longer applies. That translation engine will require some proprietary rules or code to manipulate the final GUI, however, in most modern GUI tools, that capability provides more than just refacing.

Where the power of a translation engine stands out is in the final result. To answer the question, looksoftware’s translation engine does read the 5250 telnet data stream as the source of the refaced applications. That includes any application - OS, custom, package - and does not require source code. If RPG Open Access is used, the translation engine source is the OA handler, where far more detailed and in depth information is available - thus improving the user experience dramatically. Enhancing the original RPG source code and using DDS well can be a major help in producing the final GUI and user experience.

Adding to the power of the translation engine for the looksoftware client and lookserver solution are scripting support, FTP, RPC, DDM, ADO and web service functionality. These are the means to go far beyond refacing and turn your modernization project into a repurposed, reengineered or restructured solution. This is where a strong modernization strategy can take your applications to a new future.

And you can’t do that in the 21st century, using screen scrapping!

Topics: GUI, Screen Scrapping