Why Optimization Modeling Should Move Beyond Using Scripts

Although I wrote my last script many years ago, I will attempt to summarize my reflections into these general comments:

1. Like programming in any language, scripts require domain knowledge, including the base language syntax and structure as well as any extensions.
2. Scripts are normally stored as a text file, which can be edited in something as simple as Notepad. This can make debugging syntax mistakes tedious and frustrating, especially if creating scripts is something you do sporadically.
3. Scripts require a certain aptitude and experience to keep good at it. Unless done frequently, it can feel slightly awkward every time you create one.

Algebraic Modeling Software and the Need for Automation

Similarly to application software, the algebraic mathematical modeling category has existed for about 30 years. The most common packages today are still the original ones: AIMMS, AMPL, GAMS, LINDO, MPL, OPL, and a few others. Their primary purpose is, and always will be, to build and use optimization models by coding algebraic equations.

Beyond development and debugging, however, forcing users to manually interact with their optimization models was never going to succeed for repetitive planning purposes — i.e. models that need to be solved monthly, weekly, daily, or even hourly.

Even repetitively doing just a few simple tasks — e.g., data import, solve, data export — was reason enough to want to automate a process. Many non-modeling reasons, including the need for flow control logic, selections, and integration with a variety of internal and external processes, made it a requirement.

Consider the case where a user wants to do these additional tasks along with data import, solve and data export:

• Preprocessing: e.g., running SSIS packages, R scripts, ETL/data feeds from ERP systems
• Looping: doing the same tasks multiple times
• If-then-else statements: conditional branching depending on certain solution values
• Adjust model data: e.g., set transfer prices between model solves
• Analyze solution results: e.g., evaluate opportunity values
• Post-processing: e.g., running SSIS packages, ETL/data feeds to ERP systems

Adding Script to Support Models

Early on, developers of the major algebraic modeling packages recognized that modelers, as a general rule, were never going to be software developers. Asking them to create their own compiled applications using an API just to automate processes seemed too demanding and onerous. Users required something simpler (with limited syntax) that didn’t require a complicated development environment.

The solution was to add scripting capability. By then, scripting languages had been around for many years and had already proven to be a crucial link needed to tie systems together that couldn’t otherwise be easily done (if at all). Put simply, scripts can be the “glue” that makes things work when nothing else will.

But for the algebraic modeling packages, using just an off-the-shelf scripting language — e.g., JavaScript, Python, UNIX, VBScript or Windows PowerShell — was never going to be enough. Users needed to sometimes reference specific model elements, or use specific modeling methods for certain tasks that were not part of the commercial scripting language. So most algebraic modeling languages adopted a particular scripting language and then set about to customize it for their own needs.

AMPL Shell and IBM ILOG Scripts

Two of the most popular scripting languages used with algebraic modeling software has been AMPL Shell and IBM® ILOG® Script, the latter which combines OPL models with other functions in a customized implementation of JavaScript.

I’m not going into much detail here, as there is good documentation available online. Both scripting languages are comprehensive; the user can do just about anything desired within the context of model support. Both have likely been used by thousands of users for a wide range of tasks.

AMPL’s language scripts are text file-based and can either be in Windows or UNIX. They will typically be run from a command prompt or using Windows scheduler or similar. Below is an example of AMPL Shell script copied from AMPL: A Modeling Language for Mathematical Programming, Copyright © 2003 by Robert Fourer, David M. Gay, and Brian W. Kernighan.

model steelT.mod;

data steelT.dat;

for {1..4} {

solve;

display Total_Profit >steelT.sens;

option display_1col 0;

option omit_zero_rows 0;

display Make >steelT.sens;

display Sell >steelT.sens;

option display_1col 20;

option omit_zero_rows 1;

display Inv >steelT.sens;

let avail[3] := avail[3] + 5;

}

In contrast, IBM ILOG’s scripts are embedded directly in the OPL model file. This example was copied from IBM’s website:

main {

var status = 0;

thisOplModel.generate();

 

var produce = thisOplModel;

var capFlour = produce.Capacity["flour"];

 

var best;

var curr = Infinity;

var basis = new IloOplCplexBasis();

var ofile = new IloOplOutputFile("mulprod_main.txt");

while ( 1 ) {

best = curr;

For both examples, it seems obvious that aptitude and training is required.  And keep in mind these snippets are quite short — scripts can require many more lines if complexity is needed. It might not be equivalent to writing compiled code with a language like C++, but neither is it intuitive to someone uneducated. Even someone experienced with one of these particular scripting languages might be challenged to debug and support someone else’s work.

It’s worth noting that a few vendors, like AIMMS and IBM, have recently attempted to address some of the long-recognized shortcomings from using scripts. They’ve taken steps to provide users with “advanced” tools, usually positioned as hosted and managed cloud services. Users simply need to upload their model files and push a few buttons. But traditional algebraic modeling software are unlikely to ever offer substantially more than they do today simply because of the nature of the original intent and design.

Enterprise Optimizer® Server: A Non-Programmatic Approach

In 2009, River Logic began a new product development effort to supplement its longtime prescriptive analytics modeling platform, Enterprise Optimizer® (EO). This new application was named Enterprise Optimizer Server® (EO Server).

It was intentionally designed for EO customers to automate the interactions with their EO models in a manner that did not require any programming. 

Using a management studio interface similar to Microsoft’s SQL Server Management Studio, the primary function is for users to define jobs containing tasks, which can be dragged-and-dropped onto the main job window.  A job can be for any reason, and there is no restriction to the number of jobs or number of tasks within a job that can be defined. All data is stored in a SQL database.  Here’s a sample screenshot:

Using the EO Server Management Studio (or its API), a credentialed user can easily create a job to execute these steps:

1. Create a new job: e.g., “Monthly planning run”
2. Add tasks: e.g., OpenModel, ImportAll, Solve, ExportAll
3. Change parameters: e.g., Solver, SolveTime, etc.
4. Submit job
5. Analyze messages: e.g., “CPLEX: Integer solution # 15 found. Record = -274830762.2675 Iteration: 3419  Gap: 0.00%  Branch: 0%  Branching depth: 0  Covered: 0%  Solve time: 0:01”

Then, by adding counter, logical flow control and other tasks, complex jobs are easily created that can evaluate variables and branch depending on the values. This makes running Monte Carlo and other batch processes simple and efficient. Also, jobs can be distributed across multiple servers and run in the cloud, thus being extremely scalable.

Closing Remarks

Scripting languages created long ago to support algebraic modeling languages can be very powerful.  Trained users can effectively use these scripting languages to execute just about any method necessary to read/write data, manipulate model elements, create loops with IF…THEN…ELSE statements, and so on.  But there are drawbacks: they require experience with specific syntax, are frequently stored in simple text based files, and don’t lend themselves to advanced user interfaces for debugging. Using such scripts in a secure, multi-tenant distributed environment, where each user needs a customized script, can be challenging to implement and support.

With the capabilities of River Logic’s Enterprise Optimizer® Server — including its visual, drag-and-drop style management studio interface and SQL database — modelers using old-style algebraic scripting methods should ask themselves if it’s worth continuing with a technology created decades ago; especially considering the current challenges of deploying their models in a distributed environment, across servers, or in the cloud.