This discussion is limited to the particular Monte Carlo method of randomly generating stochastic input data based on a probabilistic distribution function with parameters; solving a mathematical model using optimization algorithms; and then repeating a sequence of process steps ‘n’ times, as desired. This might sound complicated, but trust me it’s not too difficult.
For context, the following example uses River Logic’s prescriptive analytics platform, which includes two interoperable components: Enterprise Optimizer and EO+.
Enterprise Optimizer is the underlying visual modeling and optimization component. EO+ is the cloud-based component with which business users interact to manage data, generate and run scenarios, and collaboratively analyze those scenarios with Power BI embedded.
Read more about EO+ in this blog post: Next Generation Scenario Analysis in River Logic's EO+
The discussion below takes a deep dive into Enterprise Optimizer and it’s sister component, EO Server. Used by analysts and consultants, the Visual Modeler component allows user to create constraint-based optimization models with a drag-and-drop interface. At the conclusion, an example on how this approach can be embedded within a commercial application is provided.
Example: Nutmeg Candy Company
This Monte Carlo example is based on a simple EO model created many years ago. The fictional business, Nutmeg Candy Company, buys ingredients and makes and sells two products, nutcrunch and megacrunch. EO users who have taken the online training know this model well. Here is the process diagram as defined in the EO model.
Inside the Sales object, product prices are set to stochastically generate new random prices prior to each time the model is solved.
Both prices use a Normal distribution; a mean equal to the base input price (40 and 30, respectively); and a standard deviation of 7.5 for nutcrunch and 5.0 for megacrunch.
Additional data structures were added to the model schema to support the Monte Carlo process. A user-defined variable, _scenarioID, was created to store the current scenario identifier; and two user-defined tables, SolutionResults and SalesActivityMTP, were created to store each scenario solution’s results. And, a set of queries were created to manage deleting old data and to populate these tables with new data after each solve. More on this below.
The model was then saved with file name Sweet_stochastic.cor.
Enterprise Optimizer Server – Overview
In a previous blog, Why Optimization Modeling Should Move Beyond Using Scripts, I briefly introduced why Enterprise Optimizer Server, River Logic’s code-free deployment application for Enterprise Optimizer®, is a significant advancement over the ‘traditional’ algebraic modeling tools reliance on scripting languages. The following example is intended to prove why this is the case. Although it’s possible that the same example could be recreated in an algebraic modeling software program, it would take vastly more work to code, debug and deploy.
For background, EO Server:
- Supports developers by allowing them to create EO-based planning solutions with 64-bit Windows service (agent) architecture using N-servers distributed on-premise or public/private cloud.
- Includes a Server Management Studio (EOSMS) user interface in which users log in to a secure environment to create, manage and execute jobs. In the current version 4.1, there are 54 different task types available for actions like opening an EO model, setting connection string properties, defining job variables, importing data, solving a model, exporting data, and running PowerShell tasks. Jobs can be for any purpose, and there are no restrictions on the number of jobs or number of tasks within each job. Depending on the install configuration and permissions, a job can also execute processes on remote servers, thus tying together different business processes into a holistic solution.
Enterprise Optimizer Server – Controlled Loops
Automating the Monte Carlo process requires two separate but related jobs, one to act as a control loop with conditional logic needed for branching, and one to act as a master job that contains the steps that each solve process will follow. We begin with the first EO Server job, Sweet_stochastic_control, intended to control the overall process. It is important to remember that after each task is executed, the job flow can branch to any other task anywhere in the job. Generally, there are two branching choices — if the task was successful upon execution, or not successful (a small number have more than 2 choices or no choices).
Therefore, jobs that appear linear when viewing the task list in the EOSMS can instead have nested loops defined. As you will see, loops are essential when the same set of tasks must be executed repeatedly, as is the case when doing Monte Carlo simulation. They can be as simple as 2 or 3 tasks, or include as many as 50 or more tasks.
Reasons why a loop might be needed in a job include (highlighted text is included in this example):
- Stochastically defined data exists inside the EO model (to be generated for each solve).
- Scenario-dependent data to be imported from EO Server (e.g., scenarioID), a relational database, Excel or other source.
- Transfer prices or other costs/revenues to be calculated and/or modified based on the prior solution’s values or calculated unit costs, which will be used in a subsequent solve.
- Solver parameters, like the integer optimality tolerance or maximum number of integer solutions, to be adjusted based on a previous solution results.
- The prior solution’s solution results are stored in user-defined tables using action queries.
- Many more…
Enterprise Optimizer Server – Conditional Branching
At any point in the loop, but typically at the end, the condition of one or more variables can be considered. Depending on the result, the job flow is then branched upward to an earlier “beginning” task (i.e., repeat the loop) or continues to a later task. It might be helpful to think of this logic as similar to “IF…THEN…GOTO” logic in a traditional programming language, except no coding is required. And, there is no maximum number of conditions that can be evaluated, including (highlighted text in this example):
- Has the maximum number of scenarios been reached?
- Has the model reached integer optimality or some other cutoff criteria?
- Many more…
Monte Carlo Example – Defining the Control Job
Below is a screenshot of the control job’s task list, as viewed within the EO Server Management Studio user interface. Defining this job was a process of creating new tasks by dragging and dropping the 15 task types from the Toolbox in the upper-right-corner to the middle window, and then adding task parameters and other information in the window on the bottom-right-corner. No code writing was needed to successfully create this job!
Some brief task descriptions (by task number):
#1 – SetJobVariable task defines variable scenarioID and initializes it to 1.
#2 – SetJobVariable task defines variable maxSolves and initializes it to 1000.
#3 – OpenModel task opens EO model Sweet_stochastic.cor.
#4 and #5 – ExecuteQuery tasks delete old data from user-defined tables in the EO model.
#6 – After deleting old data, model is saved using the SaveAsFile task.
#7 – WriteMessage task is used to write custom message at the beginning of each loop.
#8 – Delay task is set to 5 seconds to make certain that the prior scenarioID’s solve is complete prior to the next solve starting. Note, the decision to execute cloned jobs in sequence was strictly user choice, as EO Server allows up to 999 simultaneous jobs using the same EOS agent.
#9 – DeleteJob task deletes old job with the same scenarioID, if it exists.
#10 – CloneJob task clones the master job and renames the cloned job to include the scenarioID. More on this below.
#11 – All job variables are passed from the control job to the cloned job. In this example, only scenarioID is needed in the cloned job.
#12 – The cloned job is submitted for execution by the EOS agent.
#13 – The Counter task increments the scenarioID value by +1.
#14 – The ConditionalJump task evaluates the scenarioID value and determines if it is less than or equal to the maxSolves If true, the process goes to or “jumps” to task #7, WriteMessage, and the next iteration begins. If false, it means that the maximum number of solves has been reached and the process goes to the next task #15, Exit, and the job will finish.
Monte Carlo Example – Defining the Master Job (To be Cloned During Each Loop)
The master job, Sweet_stochastic_master, to be cloned during each loop is displayed below. It was created in the same manner as job #1.
Some brief task descriptions (by task number):
#1 – OpenModel task opens EO model Sweet_stochastic.cor. This is the same model used in the control job, except it will now be open during each scenario. Note, the name could have been passed as a job variable from the control job.
#2 – WriteMessage task writes the scenarioID in a custom message. This can be helpful from separating one solve’s messages from another.
#3 – SetUserDefinedVariable stores the scenarioID value to the user-defined object variable, _scenarioID, inside the EO model. This value is referenced by the queries to properly store the solution results after each solve.
#4 – The EO model is solved using the Solve step. Note, EO Server has many task types which allow for setting the solver name (“CPLEX”, “MIO”) and a list of solver parameters.
#5 and #6 – ExecuteQuery tasks copy the _scenarioID plus model solution results from pre-defined EO tables to the user-defined tables.
#7 – SaveAsFile task saves the EO model after each solve. Notice that the same model file is being saved during each scenarioID. This parameter could be easily modified to change the model file name to something like Sweet_stochastic_{$scenario$}.cor, which would have created and saved 1,000 different model files – one for each scenarioID.
#8 – Exit task automatically closes the job.
Running the Monte Carlo Simulation
For this example, only the control job Sweet_stochastic_control was manually submitted in the EOSMS – and only just 1 time!
The job was immediately picked up by an available EOS agent and processed, which took about 95 minutes to execute 8,528 tasks or approximately 5.7 seconds per job execution. (Most time for each job was to generate and solve the model.)
Control Job messages
As displayed below, the control job resulted in over 16,000 messages generated and stored—most related to task execution—for the 1,000 times the EO model was solved.
Master Job messages
The master job stored all the messages generated during each cloned job’s execution. Over 49,000 total messages were generated, including all the solve-related messages that would normally be displayed in the EO model’s Run window during a manual solve process.
Analyzing the Results
Once the EO Server control job is complete, the Monte Carlo simulation is done. Results can be analyzed either within the EO model, such as the EO dashboard displayed below, or data can be easily exported from EO to any relational database, MS Access, MS Excel, flat file or other format.
Embedding into Applications
Importantly, both Enterprise Optimizer® and EO Server were designed to be embedded as components into other applications. The EO Server jobs described above can run strictly in memory. All job variables can be passed to the job from external applications and actions, such as submitting a job, retrieving messages, displaying results, etc. can be easily accessed the extension API, which can be called from Microsoft development tools (C++, .NET), web services, or other methods.
River Logic has a long history of working with its consulting partners, VARs, system integrators, software companies, and other companies to successfully incorporate these types of capabilities into their applications. A very good example is the Trade Promotion Optimization (TPO) solution developed with partner AFS Technologies.
Concluding Remarks
This example highlights these important concepts:
- Enterprise Optimizer models can have input data randomly generated via an easy-to-use data table editor (i.e., no coding). Stochastic data only requires selecting a statistical distribution from the available list and then defining the necessary parameters.
- EO Server has an easy-to-use Server Management Studio user interface, in which job(s) can be defined in to automate the necessary sequence of tasks. With 54 different task types, EO Server can assist a wide range of users with differing requirements by interacting with their EO models and connecting various servers and processes within a network to develop holistic applications.
- Both EO and EO Server can be embedded as components into applications, which can be used for a wide range of use cases including Monte Carlo optimization.
Finally, one of the key benefits with the River Logic platform, including the Visual Modeler as detailed above, is that it’s built on Microsoft Azure, adding scale, elasticity of services, security, and computer power. It also offers cutting-edge, easy-to-deploy capabilities like machine learning, predictive modeling, IoT, and more.
