Ensemble Size and Speed Benchmarking

Ensembles are specifically designed for optimal usability, memory usage, and computational speed. In this tutorial we explore the size and speed related characteristics of Ensembles compared to using the equivalent individual models. We aim to begin to answer the following questions: - How much memory does an ensemble use when working with it compared to working with the equivalent individual models? - How much disk space is used to store ensembles compared to the equivalent individual models? - How long does it take to run FBA for all members of an ensemble compared to the equivalent individual models?

Ensemble memory requirements during use and when saved

Ensembles are structured to minimize the amount of memory required when loaded and when being saved. One of the major challenges when working with ensembles of models is having all of the models readily available in memory while conducting analyses. With efficient packaging of the features that are different between members of an ensemble, we were able to significantly reduce the amount of memory and hard drive space required for working with ensembles of models.

In [1]:
import sys
import os
import psutil
import medusa
import numpy
from medusa.test import create_test_ensemble
In [2]:
# RAM required to load in a 1000 member ensemble

# Check initial RAM usage
RAM_before = psutil.Process(os.getpid()).memory_info()[0]/1024**2 # Units = MB

# Load in test ensemble from file
ensemble = create_test_ensemble("Staphylococcus aureus")

# Check RAM usage after loading in ensemble
RAM_after = psutil.Process(os.getpid()).memory_info()[0]/1024**2 # Units = MB
RAM_used = RAM_after - RAM_before
# Print RAM usage increase due to loading ensemble
print("%.2f" % (RAM_used), "MB")
57.82 MB
In [3]:
# The test S. aureus model has 1000 members
print(len(ensemble.members),'Members')
1000 Members
In [4]:
# RAM required to load a single individual model

from copy import deepcopy
# Check initial RAM usage
RAM_before = psutil.Process(os.getpid()).memory_info()[0]/1024**2 # Units = MB

# Deepcopy base model to create new instance of model in RAM
extracted_base_model_copy = deepcopy(ensemble.base_model)

# Check RAM usage after loading in ensemble
RAM_after = psutil.Process(os.getpid()).memory_info()[0]/1024**2 # Units = MB
RAM_used = RAM_after - RAM_before
# Print RAM usage increase due to loading ensemble
print("%.2f" % (RAM_used), "MB")
17.50 MB
In [5]:
# If we were to load the individual base model as 1000 unique
# model variables we would use 1000x as much RAM:
RAM_used_for_1000_individual_model_variables = RAM_used * 1000
print("%.2f" % (RAM_used_for_1000_individual_model_variables), 'MB or')
print("%.2f" % (RAM_used_for_1000_individual_model_variables/1024.0), 'GB')
17500.00 MB or
17.09 GB
In [6]:
# Pickle the ensemble and extracted base model
import pickle
path = "../medusa/test/data/benchmarking/"
pickle.dump(ensemble, open(path+"Staphylococcus_aureus_ensemble1000.pickle","wb"))
pickle.dump(extracted_base_model_copy, open(path+"Staphylococcus_aureus_base_model.pickle","wb"))
In [7]:
# Check for file size of ensemble
file_path = "../medusa/test/data/benchmarking/Staphylococcus_aureus_ensemble1000.pickle"
if os.path.isfile(file_path):
    file_info = os.stat(file_path)
    mb = file_info.st_size/(1024.0**2) # Convert from bytes to MB
    print("%.2f %s" % (mb, 'MB for a 1000 member ensemble'))
else:
    print("File path doesn't point to file.")
6.67 MB for a 1000 member ensemble
In [8]:
# Check for file size of extracted base model
file_path = "../medusa/test/data/benchmarking/Staphylococcus_aureus_base_model.pickle"
if os.path.isfile(file_path):
    file_info = os.stat(file_path)
    mb = file_info.st_size/(1024.0**2) # Convert from bytes to MB
    print("%.2f %s" % (mb, 'MB per model'))
else:
    print("File path doesn't point to file.")

print("%.2f" % (mb*1000),'MB for 1000 individual model files.')
print("%.2f" % (mb*1000/1024),'GB for 1000 individual model files.')
1.07 MB per model
1070.01 MB for 1000 individual model files.
1.04 GB for 1000 individual model files.

Flux analysis speed testing

Running FBA requires a relatively short amount of time for a single model, however when working with ensembles of 1000s of models, the simple optimization problems can add up to significant amounts of time. Here we explore the expected timeframes for FBA with an ensemble and how that compares to using the equivalent number of individual models. It is important to note that during this benchmarking, we assume that the computer being used is capable to loading all individual modelings into the RAM; this may not be the case for many modern laptop computers (e.g., ~16GB spare memory required).

In [9]:
import time
from medusa.flux_analysis import flux_balance
In [10]:
# Time required to run FBA on a 1000 member ensemble using the innate Medusa functions.
runtimes = {}
trials = 5
for num_processes in [1,2,3,4]:
    runtimes[num_processes] = []
    for trial in range(0,trials):
        t0 = time.time()
        flux_balance.optimize_ensemble(ensemble, num_processes = num_processes)
        t1 = time.time()
        runtimes[num_processes].append(t1-t0)
    print(str(num_processes) + ' processors: ' + str(numpy.mean(runtimes[num_processes])) + ' seconds for entire ensemble')
1 processors: 87.24728102684021 seconds for entire ensemble
2 processors: 44.09945402145386 seconds for entire ensemble
3 processors: 32.84902577400207 seconds for entire ensemble
4 processors: 27.70060839653015 seconds for entire ensemble
In [11]:
# Time required to run FBA on 1000 individual models using a single processor.
# This is the equivalent time that would be required if all 1000 models were pre-loaded in RAM.

trial_total = []
for trial in range(0,trials):
    t_total = 0
    for member in ensemble.members:
        # Set the member state
        ensemble.set_state(member.id)
        # Start the timer to capture only time required to run FBA on each model
        t0 = time.time()
        solution = ensemble.base_model.optimize()
        t1 = time.time()
        t_total = t1-t0 + t_total
    print("%.2f" % (t_total) ,'seconds for 1000 models')
    trial_total.append(t_total)
print("%.2f" % (numpy.mean(trial_total)) ,'second average for 1000 models')
35.06 seconds for 1000 models
34.51 seconds for 1000 models
34.49 seconds for 1000 models
34.62 seconds for 1000 models
34.37 seconds for 1000 models
34.61 second average for 1000 models

Using individual models stored in memory is faster than an equivalent ensemble with 1-2 processors, but Medusa is faster with an increasing number of processors. Keep in mind, however, that this comparison doesn’t consider the time it takes to load all of the models (~200x faster in Medusa for an ensemble this size), make any modifications to the media conditions for an ensemble (one operation in Medusa; 1000 independent operations with individual models), and that using individual models requires far more memory (~300x in this case).

This comparison also doesn’t factor in the time required for the first optimization performed with any COBRApy model. When a model is optimized once, the solver maintains the solution as a starting point for future optimization steps, substantially reducing the time required for future simulations. Medusa intrinsically takes advantage of this by only using one COBRApy model to represent the entire ensemble; the solution is recycled from member to member during ensemble FBA in Medusa. In contrast, the first optimization step for every individual model loaded into memory will be more computationally expensive, as seen by the timing in the cell below.

In [13]:
# Time required to run FBA on 1000 individual models with a complete solver reset
# before each optimization problem is solved.

# Load fresh version of model with blank solver state
fresh_base_model = pickle.load(open("../medusa/test/data/benchmarking/Staphylococcus_aureus_base_model.pickle","rb"))
# Determine how long it takes to run FBA on one individual model
t0 = time.time()
fresh_base_model.optimize()
t1 = time.time()
t_total = t1-t0
# Calculate how long it would take to run FBA on 1000 unique individual models
print("%.2f" % (t_total*1000), 'seconds for 1000 models')
192.96 seconds for 1000 models