The Power of Data Simulation

Using Python’s NumPy Random Generator to Model Randomness & Build Intuition

Simulating data is an indispensable tool in data science. This paper uses website conversion as an example to simulate large datasets, build intuition for the randomness in rate-based metrics, and verify results against the Normal approximation.

Introduction

Simulating data is an indispensable tool in the business and data science community. It offers a cost-effective way to model complex relationships and rare events, enabling exploration of various scenarios without real-world consequences.

Simulations can:

• Measure competing methods’ performance on shared datasets
• Assess how statistical models perform when ground truth is known
• Refine analytical methodologies
• Support training of machine learning models in the absence of real data

Using website conversion as our example, we simulate huge datasets and build intuition for the randomness we might expect to see.

---

Website Conversion Example

Imagine your website converts 10% of visitors to paying customers. Over 10,000 visitors:

\[10000 \, \text{visitors} \times \frac{10 \, \text{conversions}}{100 \, \text{visitors}} = 1000 \, \text{conversions}\]

How much could that number vary on any given day?

---

Simulating Data

Simulate 1,000,000 experiments with \(n = 10{,}000\) and \(p = 0.10\):

import timeit
from numpy.random import binomial, seed

seed(10)

conversion_rate = 0.10
number_of_visitors = int(1e4)
number_simulated_experiments = int(1e6)

start = timeit.default_timer()
control_simulated_results = binomial(
    number_of_visitors,
    conversion_rate,
    size=number_simulated_experiments,
)
stop = timeit.default_timer()
print(f"simulation runtime: {round(stop - start, 3)} secs")

simulation runtime: 0.03 secs

Graph Simulation Results

At the end of each simulation, record and plot how many conversions took place. With enough simulations, website conversion forms a predictable bell-shaped pattern.

import pandas as pd
import plotly.express as px

df = pd.DataFrame(control_simulated_results, columns=["value"])
fig = px.histogram(df, x="value", nbins=60, title="Distribution of Simulated Conversions")
fig.show()

Use Simulation Results to Estimate Randomness

1. What’s the chance of 950 conversions or fewer?
2. What’s the chance of 1050 conversions or more?
3. What’s the chance of between 950 and 1050 conversions (exclusive)?

from numpy import where, mean

prob = mean(
    where(
        (control_simulated_results > 950) & (control_simulated_results < 1050),
        1,
        0,
    )
)
print(f"{prob:.2%}")

90.10%

Verify Simulation Results

Because rate-based metrics are normally distributed, verify with the Normal approximation:

\[F(0.095) = \Phi\left( \frac{0.095 - 0.10}{0.003} \right) = \Phi\left( -1.67 \right) = 0.0478\]

Type	Probability
950 conversions or less	4.78%
1050 conversions or more	4.78%
Between 950 and 1050 conversions	90.44%

Slight differences are explained by the continuous nature of the Normal approximation. Our data is discrete.

---

Conclusion

By simulating 1MM experiments we observed patterns and trends that would be difficult or costly to capture in real-world scenarios. You now know how to use NumPy to generate rate-based simulated data and interpret the results.

This approach is even more valuable when dealing with problems that lack closed-form analytical solutions.

---

Upcoming Topics

• Experiment Design & Analysis with NumPy Random: P-values and Statistical Power
• Continuous Metrics
• Sequential & Bayesian Testing
• Using Prior Information to Improve Effect Size Estimates
• CUPED: Using Covariates to Reduce Experimentation Run-time

---

Big THANK YOU to my colleague Joseph Powers, PhD, who introduced me to Quarto and simulation studies.