White Paper:
Real Time Enterprise

Overview
Conventional Information Flow
Real Time Information Flow
Real Time Data Warehouse
Software Compatibility
Business Process Management
Return on the Investment

Overview

The Real Time Enterprise (RTE) proposes that gathering and using information in near real time will benefit some parts of the business.  A classic example is presenting custom browser screens to a customer looking at a website, based on the customer's previous buying patterns, their current click stream, the current inventory, and any promotional campaigns.

The RTE is a business proposition and should be considered in terms of return on investment (ROI) -- what will be the return and what will it cost to obtain it.  There are at least three significant factors to consider and overcome to achieve the RTE:

• Data Warehouse:  The process of cleaning, transforming, and consolidating data from multiple process steps is known as data warehousing.  Most data warehousing occurs on a time scale of hours or days, but the RTE requires this data on a time scale of minutes or even seconds.  There may be substantial technical and procedural changes needed to achieve this.

• Software Compatibility:   The RTE implies rapid communication between process steps that can be different software applications.  For example, the data warehouse might talk to the Customer Relationship Management (CRM) software, and this talks to the inventory software and the web server.  These are often incompatible software "silos" that will have to communicate quickly and easily for the RTE to succeed.

• Business Process Management:  The example of building custom browser screens requires assimilating data from multiple sources and making complex decisions in only seconds.  This requires Business Process Management/Workflow Management (BPM/WfM) software that models the process and automates the decision steps. 

These obstacles are balanced by an impressive set of returns that can often justify the investment:

• Real Time Execution:  Decisions are made based on current data while the process is executing.

• Consistent Execution:  An automated system provides a complete and consistent process execution every time.

• Best Practices:  The best practices and expertise of company experts are encapsulated and automated in the BPM/WfM.  The system always makes the best possible decision .

• Process Documentation:  Building a BPM/WfM model requires understanding and documenting the process.  A company can demonstrate they understand their process and can meet compliance requirements such as are required by the FDA or Sarbanes-Oxley regulations.

Conventional Information Flow

Every company has processes and procedures.  The figure below illustrates how process information (red arrows) flows  and is utilized in the conventional (non-real time) company.  Whether the company makes strawberry jam, builds tractors, or processes loan applications, there is a process consisting of steps.

Each step can produce data that can be in different formats and databases.  It is easier to work with this data if it is cleaned, converted to standard units, and consolidated in one database.  This activity is called data warehousing.  Once the data is in a more manageable form in the data warehouse, it can be analyzed and reports prepared.  Reports are then considered by the company.  Ways are discovered to improve the process and action is taken, usually by means such as editing the procedure manual and the computer software.

Information Flow in a Conventional Company

The next figure below suggests time scales for information flow and process change in the conventional (non-real time) company.  The individual steps in the process itself are often very quick and may take only seconds.  Loading updates into a data warehouse is frequently done overnight, giving it a moderate time scale of hours up to a day.  The data analysis may be driven from the data warehouse uploads or may occur on an infrequent ad hoc basis.  Reviewing reports and taking action to change the process can take days, weeks, or months.

Information Timing in a Conventional Company

It should be clearly noted that there is absolutely nothing wrong with the above information flow and process change methodology.  This is the way most business function, and it serves them well in many circumstances.  The point to be made is that the information does not flow back to modify the business process in real time.  Implementing real time systems does not imply that any part of the existing system will be replaced or eliminated.  The real time enterprise adds current information feedback coupled with automated decision tools to provide real time control of some parts of the business process.

Real Time Information Flow

The figure below illustrates information flow in the real time enterprise.  Note that analysis, reporting, and review have been replaced by the "business process model."  This BPM/WfM model is constructed after careful analysis of the process steps and the way information guides decisions and process flow.  The BPM/WfM model is driven by real time data and is able to provide detailed control within individual process steps in real time.

Data to drive the BPM/WfM model is derived from the data warehouse.  All of the motivations to have a data warehouse are as valid in the the real time scenario as they are in the conventional business information flow model.

Information Flow in the Real Time Enterprise

The next figure below suggests time scales for information flow and process change in the real time enterprise.  The individual steps in the process are themselves unchanged between the conventional and RTE companies, and are often executed on a time scale of  seconds.  Some of the updates in the data warehouse must now also be done on a time scale of seconds to provide current data to feed the real time decision process.  Other data from the data warehouse such as trends and marketing campaigns would by loaded at a moderate time scale from hours up to a day.  Data from the warehouse fuels the real time decision process in the BPM/WfM model which then feeds control information into the individual process steps as they are executing.

Information Timing in the Real Time Enterprise

Real Time Data Warehouse

Data warehouses gather data from several applications and databases, transform the data into comparable units, perform quality checks such as detecting outliers, and load data into one common repository.  One motivation for the data warehouse is efficiency since data preparation would have to be done by each analysis using the data and it makes sense to do the prep work once and store it for general use.  The data warehouse also provides one consistent view of the data containing good quality data, stored in an accessible form intended to support analysis.  Data loads into the warehouse are periodic, allowing the data to remain stable the rest of the time.  All of these factors remain valid when applied to the RTE.

The traditional data warehouse loads new data onto disk drives on a daily, weekly, and monthly basis.  These loads are often large and many warehouses load over 100 million new rows each  month.  This is rather different from the needs of the RTE where loads may only be a few rows but need to occur with great frequency on time scales of seconds and minutes.  Many of the traditional data warehouse products are not ideally suited to this requirement.  Existing software and procedures may need to be modified.  Rather than writing to disk, the initial high speed and low volume data storage may best be written into computer memory.

Software Compatibility

Data warehouses imply reading data from several software applications and databases.  The process steps may themselves be different applications, and controlling them in real time implies also rapidly writing to these applications.  Since the process might be distributed over a network, it may be necessary to read and write across different hardware, operating systems, applications, languages, and databases.

Middleware are software packages that allow incompatible applications to talk with each other.  In the figure below, three middleware packages are used to interconnect the Inventory, CRM, and data warehouse applications.  Although this does work, the complexity of managing multiple middleware packages grows as the required number of interconnections increases.

Middleware Connecting Individual Applications

The next figure below shows interconnecting applications using a common data bus.  The bus hub usually maintains its own database, and it is only necessary to interface this with each individual application.  This is much simpler to maintain than a system with interconnects between each pair of applications.  The central hub is also a good location to embed a high speed memory store to support the real time components of the data warehouse.

Common Data Bus Connecting Individual Applications

The problem of interconnecting applications has lessened in recent years.  XML standards have emerged that support interactions and data exchange, and many of the major application and database vendors today have products that support XML.  Similarly, many vendors today also provide Java™ interfaces and drivers that allow user-written Java programs to directly access the vendor's products.

Business Process Management

The Business Process Management/Workflow Management (BPM/WfM) module is where the real innovation in the RTE occurs.  The BPM/WfM module is a software application that is configured to model the business process.  When driven by real time data, the BPM/WfM module delivers real time decisions and control into the business process steps.  BPM/WfM applications can model a variety of process, such as making strawberry jam, building tractors, or approving loan applications.

One way of thinking about the BPM/WfM module is to imagine taking the two or three people who best understand the company's business process and then extracting all their knowledge and combining this in an automated computer system.  The next time the business process is executed, these experts are in the computer monitoring data and making decisions.  This expertise becomes available 24/7 to provide an automated consultation.

There are in fact a number of ways to model business processes, some of which are described in the White Paper Modeling Business Processes.  We will take a simple example and briefly demonstrate a  technique called "structured objects models."  Our simple business process is as follows:  Running the MonthEndReports requires producing two files, File_A and File_B.  File_A is produced by Job_A, and Job_A cannot run until the RawData and adequate DiskSpace are available.  File_B is produced by Job_B, and Job_B cannot run until File_A, adequate DiskSpace, and adequate CPU resources are available.

The figure below illustrates business process modeling based on structured objects.  This requires identifying the major components of the problem and arranging them to capture their temporal, functional, and hierarchical order.  The result resembles a business outline, and at run time the control mechanism will read the outline in a way similar to how a human would read it.  Multiple read cycles can occur if needed to complete the task.  Each label in the outline contains both data and code that can gather information, make decisions, and invoke actions.

The labels are objects and provide full compatibility with modern object-oriented analysis, design, and programming.  Structured objects views the problem at a higher abstraction level than rules or flow charts and hides much of the complexity within the labels.  These models are easily scaled and provide a good project overview.

Return on the Investment

The RTE is about having all the current data and applying it consistently using best practices and expert knowledge.  It is an automated system that can be deployed where needed.  Unlike conventional computer programs, the RTE technology captures and represents knowledge in a way that is accessible to management and business unit staff.

The primary purpose for establishing a RTE environment is to deliver real time feedback to a business process.  One example was to present custom browser screens to a customer looking at a website.  Driving a BPM/WfM module with data on the customer's previous buying patterns, their current click stream, the current inventory, and any promotional campaigns results in the optimal screens and the best results from the customer interaction.

This approach makes sense for almost any process.  Consider a pharmaceutical  production process.  It's 2:00 AM and a technician is looking at the current run status.  Something needs to be done - should we add glucose - one or two liters?  Maybe the tank temperature should be increased one degree?  Making the correct decision will affect the quality and quantity of the run and will have significant economic consequences.  If this process is part of an RTE system, the current run data as well as data from the last 20 runs on this production line are immediately available from the data warehouse.  The knowledge from the scientists and engineers who built the process has been captured in the BPM/WfM module and is available for an online consultation.  The technician invokes the RTE system against this process step and is guided to take the correct actions.

Building the process model in the BPM/WfM module results in the process being fully understood and documented.  This exercise frequently uncovers the fact that no one completely understood the process, and companies often find that the best previous documentation was a dog eared procedure manual full of marginal scribbles and sticky notes.  Process investigation often results in discovering process improvements.  Process documentation in the RTE is of value in meeting requirements such as from HIPAA, Patriot Act, or Sarbanes-Oxley.

Capturing best practices and expert knowledge from subject matter experts within the company can be an effective hedge against these individuals leaving the company.   Once automated, their expertise can be used where needed for automated consultations.

The new documentation in the BPM/WfM module is accessible by both computers and business unit staff.  The documentation of the process and the expression of the process in a computer become one and the same, reducing problems with updating and synchronizing process and process documentation.  Similarly, the continuous changes that all processes undergo are entered directly into the system and go into immediate effect, reducing problems with retraining staff to know and comply with procedures.

The RTE should be viewed as a return on investment proposition.  It will have costs in both technology and behavioral changes, but it is can provide excellent returns when carefully applied.