The evolution of workflow

The IT industry is known for its buzzwords - concepts that promise to be the panacea for all business issues. Each of them captures the imagination of the industry for a while but then gradually fades into the background, soon to be forgotten.

Workflow, however, does not fit into this category, emphasises Mark Ehmke, MD of Staffware in SA. "Despite some analysts predicting the demise of workflow a few years ago, it has morphed several times and taken on various forms and labels.

"It has earned its place in the world of technology and has become known as a discipline, a practice and a concept," he says.

Workflow started in the 1980s in two forms. Firstly, it was a routing mechanism for moving document images within a document management system (DMS), but was also a means of routing financial documents within financial systems. Staffware was the first to create an independent workflow system by splitting the workflow from the financial system.

"Since these documents were similar in nature, the two product sets would often compete for document-based workflow projects. The independent workflow vendors would typically team up with other DMS vendors in a `best of breed` approach with well-defined integration toolkits between their products," Ehmke explains.

Many smaller imaging (scan and retrieval) companies then introduced rudimentary workflow for ad-hoc routing of their images within their products. These non-scalable workflows caused the introduction of the phrase "production workflow" to distinguish the larger systems from these smaller ones. This caused Gartner to coin the "production", "administrative" and "ad-hoc/collaborative" categories which were used extensively by vendors and users alike to describe the different markets.

The 1990s saw a typical consolidation phase within the workflow industry with many workflow vendors unable to compete in the generic workflow space. Their solution was to focus on niche markets such as call centres or to be bought out by application vendors. Staffware remained the dominant player, successfully deploying many enterprise workflows, and becoming recognised as an enterprise infrastructure product.

Ehmke explains that application vendors, such as SAP, introduced their own brand of workflow into their applications over the same period, and often didn`t even refer to it as workflow. This led to Gartner changing its categories to "enterprise", "application" and "collaborative" workflows.

In the meantime, many of the DMS vendors, like FileNet and Eastman, began to produce integrated document management (IDM) of tightly integrated workflow within their DM suite, which resulted in a defocus on process-specific workflows and an increased focus on document specific workflows.

"The e-commerce strategies of the two camps differed enormously," Ehmke says. "The DMS vendors positioned themselves as knowledge portals, while the enterprise workflow vendors positioned themselves as the process automation or `missing link` between a company`s channels to market and its back office systems.

"Now, in the new millennium, enterprise workflow is seen as a critical layer within the `intelligent business` model. Enterprise workflows no longer compete with the DMS products, but rather complement the middleware/messaging/Enterprise Application Integration (EAI) layers of a company where the primary focus is on integration of customer facing systems to the back office.

"These products, however, are inherently transaction-based rather than process-based," he explains.

Analysts have now identified a "Business Process Management" (BPM) market as one that encompasses human-to-human, human-to-system and system-to-system business process automation within the intelligent business model. As happened in the workflow industry, a major consolidation is again happening in this new market. Many EAI vendors, including BEA and Candle, are partnering enterprise workflow vendors like Staffware to address this space, while others are writing their own BPM or are acquiring niche workflow vendors.

Taking all this history into account, there remain several types of workflow and choosing one depends on what you want to achieve with the technology. Many large organisations use more than one workflow product supplied by different companies and the same workflow product can even be used in a number of different ways.

The most common segmentations are:

The key goal of enterprise workflow is to manage large numbers of similar tasks and to optimise productivity. This is achieved by automating as many activities as practical, and to relentlessly pursue more and more automation until Straight-Through-Processing has been achieved and human input is needed only for managing exceptions.

Enterprise workflow can manage very complex processes and can be tightly integrated with legacy systems. In fact, the trend is to embed the workflow component into large applications where its role is to act as a Rules Engine. This brings us to two further segmentations within production workflow, known as autonomous workflow engines and embedded workflow.

An autonomous workflow management system is functional without any additional application software, except perhaps database management systems and message queuing middleware. An embedded workflow management system, on the other hand, is only functional if it is employed within the surrounding system, e.g. an ERP system. The latter is valuable for users to be able to differentiate between rules-based sectors of an application that are normally activated by database triggers, and workflow engine-based components that usually allow for a more complex specification of processes.

Applications such as ERP and Financial systems require workflow as an inherent part of the supplied package in order to remain competitive.

As opposed to embedded workflow, described above, application workflow is written directly into the application itself and is so intertwined within the functionality of the product it is often not referred to as workflow but rather as the process itself. For example, supply chains, proof of delivery and procurement are all examples of workflows that are sold as key functionality of the package.

The most important feature of a collaborative workflow system is the ease to define the process. Typically, there are many definitions running concurrently and they tend to involve a large number of employees or teams sharing information towards delivering a common goal. Groups can vary from small, project-orientated teams to widely dispersed people with interests in common. Effective use of collaborative workflow to support team working is now considered a vital element in the success of enterprises of all kinds. The use of the Internet to support team communications across enterprises is also a critical success factor to most organisations.

Process definitions are usually created using forms - and if the definition is too complex for the form, then you have to use another product. Flexibility is more important than productivity, and these systems handle one or two orders of magnitude.

Ad-hoc workflow systems allow individual users to create and amend process definitions very quickly and easily to meet circumstances as they arise. It is possible to have almost as many process definitions as there are instances of the process.

Ad-hoc workflow maximises flexibility in areas where throughput and security aren`t major concerns. Whereas in production workflow, the organisation clearly owns the process, ad-hoc workflow users own their own processes.

Ehmke stresses that, despite the above classifications, there are always going to be exceptions and grey areas. "Workflow engines rarely, if ever, work alone," he concludes.