Why I Switched to Polars From Pandas

Benchmarking Polars vs Pandas DataFrames

If you use Pandas DataFrames to store and analyze large datasets, consider switching to Polars. Benchmarks across read, write, construct, pivot, join, and filter — Polars wins every time.

TL/DR;

If you use Pandas DataFrames to store and analyze large datasets, consider making a change to Polars.

I use DataFrames because they allow me to think relationally — just like I do when I write SQL. I use them extensively in virtually all of my work; app development, exploratory data analysis, and summarizing large datasets.

Check out below the benchmarks I measured for typical DataFrame operations like read, write, construct, and group by. These will convince you that a change is necessary.

System & Environment

• All benchmarks ran on a 16-inch MacBook Pro (2021) equipped with an Apple M1 Max chip and 64 GB of unified memory, running macOS Sonoma 14.7.
  
  
• Tests used Python 3.10.9, with Pandas 1.5.3 and Polars 1.33.0 installed.

Methodology

I benchmarked Polars and Pandas across common DataFrame operations using both real and simulated data.

- For read and write tests, I used a 10 MB CSV dataset from the Frictionless Data repository.
- For constructor, pivot, join, and filter benchmarks, I generated a simulated dataset to ensure consistent structure and reproducibility.
- Each operation was executed 1,000 times per library, and the mean execution time was recorded.

Read | Write

Download 10 mb file: https://raw.githubusercontent.com/frictionlessdata/datasets/main/files/csv/10mb.csv

Metric	Mean Speed Increase	Standard Deviation
Read	18.40	3.20
Write	39.41	5.85

AI Generated Summary

Polars is faster at reading and writing CSVs than Pandas because it’s built in Rust with parallelized, memory-efficient algorithms that fully utilize multiple CPU cores.

Construct | Pivot

import numpy as np
import string
import random

N = int(5e6)

def random_str(n=5):
    return ''.join(random.choices(string.ascii_letters, k=n))

data = {
    "id": np.arange(N),
    "value": np.random.randint(0, 1000, size=N),
    "flag": np.random.choice([True, False], size=N),
    "category": np.random.choice(["A", "B", "C", "D"], size=N),
    "score": np.random.randn(N),
    "year": np.random.randint(2000, 2025, size=N),
    "name": [random_str(5) for _ in range(N)],   # random short strings
    "ratio": np.random.random(size=N),
    "count": np.random.randint(1, 100, size=N),
}

Metric	Mean Speed Increase	Standard Deviation
Constructor	1.89	0.14
Group By	23.99	1.72

AI Generated Summary

Constructor: Polars DataFrame constructor is faster than Pandas because it uses Rust’s zero-copy memory model and parallel data parsing, avoiding Python’s object overhead.

Pivot Tables: Polars outperforms Pandas in pivot table operations by executing grouped aggregations in parallel and leveraging an optimized query engine built on Apache Arrow for efficient columnar computation.

Joins | Filter

Metric	Mean Speed Increase	Standard Deviation
Inner Join	2.09	0.52
Union All	1.32	0.64
Filter	1.16	0.36

AI Generated Summary

Joins (Inner + Union All): Polars performs joins faster than Pandas by using parallelized hash join algorithms implemented in Rust and operating directly on Arrow memory, minimizing copying and serialization overhead.

Filter Clauses: Polars evaluates filter expressions lazily and in parallel at the columnar level, enabling vectorized, cache-efficient filtering that significantly outpaces Pandas’ row-wise execution in Python.

Conclusion

Across the board Polars is faster at all of the typical DataFrame operations we tested. The syntax differences are minimal and AI can quickly convert your Pandas code.

If you work with large DataFrames, Polars is worth the 30-minute migration — faster runtimes, smaller memory footprint, and minimal code changes.

Code References

Operation	Pandas Example	Polars Example	Times Faster
Read CSV	df = pd.read_csv("data.csv")	df = pl.read_csv("data.csv")	18.40
Write CSV	df.to_csv("out.csv")	df.write_csv("out.csv")	39.41
Constructor	df = pd.DataFrame(data)	df = pl.DataFrame(data)	1.89
Group By	df.groupby("col")["value"].mean()	df.groupby("col").agg(pl.col("value").mean())	23.99
Inner Join	df1.merge(df2, on="id", how="inner")	df1.join(df2, on="id", how="inner")	2.09
Union All	pd.concat([df1, df2], ignore_index=True)	df1.vstack(df2)	1.32
Filter	df[df["value"] > 10]	df.filter(pl.col("value") > 10)	1.16

*** Results will vary depending on machine and size/structure of your datasets.

AI Generated Summary: Why Union All Has High Variance

Polars’ pl.concat() (or vstack()) uses multi-threaded Arrow buffer operations that cut average runtime but introduce non-deterministic variance. On macOS with M1 Max chips, the OS may schedule threads differently across performance and efficiency cores, causing some runs to finish faster or slower. In short, parallelism improves mean speed but naturally increases variance—explaining the wider spread in the “Union All” benchmark.

AI Generated Summary: Why Constructor is Reliably Faster

The constructor benchmark is the most reliable because it’s a pure in-memory, CPU-bound operation with no I/O, threading, or garbage collection noise. Each run follows the same predictable memory layout, making it a clean measure of Polars’ and Pandas’ internal efficiency without interference from system variability.