Python Polars

Table of Contents

• Introduction
• Why Polars?
• Installation
• Core Concepts
  • Data Types
  • Series
  • DataFrame
  • LazyFrame
• Reading & Writing Data
• Data Manipulation
  • Select columns
  • Filter rows
  • Add / modify columns
  • GroupBy
  • Sort
  • Join
  • Pivot / Melt
  • Window functions (over)
• The Expressions API
  • Expression building blocks
  • Named expressions and reusability
  • Contexts
• Lazy vs. Eager Execution
  • Eager API
  • Lazy API
  • Streaming mode (for datasets larger than RAM)
• Performance
• Interoperability with pandas
• Real-World Examples
  • Example 1: Parse and aggregate server logs
  • Example 2: Time-series resampling
  • Example 3: Multi-table merge with complex logic
• Best Practices and Tips
• When to Use Polars (vs. pandas vs. Spark)
• Conclusion
  • Further reading

Introduction #

Polars is a lightning-fast DataFrame library for Python, written from the ground up in Rust. Designed to handle datasets much larger than available RAM, it offers a rich expression API, zero-copy data structures, and a compelling alternative to pandas for data-intensive workflows.

While pandas has been the de facto standard for data manipulation in Python for years, its single-threaded architecture and memory-hungry operations can become bottlenecks at scale. Polars addresses these limitations head-on with a modern, multi-threaded design and a query optimizer inspired by database internals.

Why Polars? #

Key advantages:

• Speed — Polars is often 5–50× faster than pandas on common operations (groupby, joins, filters) thanks to its parallel vectorized engine.
• Memory efficiency — Arrow columnar format with zero-copy reads. You can process datasets larger than RAM using the lazy streaming API.
• Expressive API — A rich set of expressions (.col(), .sum(), .shift(), .cumsum(), .rolling(), etc.) that compose cleanly.
• No index headaches — Polars has no pd.Index — rows are identified by position, which eliminates a major source of bugs and confusion.
• Predictable performance — The query optimizer rewrites your lazy query to minimize work (predicate pushdown, projection pushdown, join order optimization).

Installation #

1pip install polars

To enable all optional features (e.g., pandas interop, xlsx, cloud):

1pip install "polars[all]"

Check your version:

1import polars as pl
2print(pl.__version__)

Polars has no mandatory Python dependencies — it ships its own Rust-compiled binary wheels for Linux, macOS, and Windows.

Core Concepts #

Data Types #

Polars leverages the Apache Arrow memory model. Key scalar types:

Series #

A Series is a one-dimensional, typed array — like a column.

1s = pl.Series("numbers", [1, 2, 3, 4, 5], dtype=pl.Int32)

DataFrame #

A DataFrame is a collection of named Series, all having the same length.

1df = pl.DataFrame({
2    "name": ["Alice", "Bob", "Charlie"],
3    "age":  [25, 30, 35],
4    "city": ["NYC", "LA", "Chicago"],
5})

LazyFrame #

A LazyFrame is a query plan that hasn't been executed yet — the core of the optimization engine.

1q = df.lazy()            # convert to lazy query
2q = q.filter(pl.col("age") > 28)
3result = q.collect()     # execute the plan

Reading & Writing Data #

Polars supports many formats natively:

 1# CSV
 2df = pl.read_csv("data.csv")
 3df.write_csv("out.csv")
 4
 5# Parquet (first-class citizen)
 6df = pl.read_parquet("data.parquet")
 7df.write_parquet("out.parquet")
 8
 9# JSON / NDJSON
10df = pl.read_json("data.json")
11df.write_ndjson("out.ndjson")
12
13# Excel (requires xlsx2csv feature)
14df = pl.read_excel("data.xlsx")
15
16# AVRO
17df = pl.read_avro("data.avro")
18
19# Delta Lake (requires deltalake feature)
20df = pl.read_delta("path/to/delta-table")
21df.write_delta("path/to/delta-table")
22
23# Database (SQL)
24df = pl.read_database("SELECT * FROM table", connection_uri)

Reading large files progressively (chunked / streaming):

1# Process CSV row-by-row without loading everything into memory
2with pl.read_csv_batched("huge.csv") as reader:
3    for batch in reader:
4        process(batch)  # each batch is a DataFrame

Data Manipulation #

We'll use a sample DataFrame for the examples:

1import polars as pl
2
3df = pl.DataFrame({
4    "id":      [1, 2, 3, 4, 5],
5    "product": ["Widget A", "Widget B", "Gadget A", "Gadget B", "Widget A"],
6    "price":   [19.99, 29.99, None, 49.99, 19.99],
7    "qty":     [10, 5, 7, 3, 12],
8    "date":    pl.date_range("2025-01-01", "2025-01-05", "1d"),
9})

Select columns #

1df.select("product", "price")
2df.select(pl.col("price").alias("unit_price"))

Filter rows #

1df.filter(pl.col("price") > 20)
2df.filter(pl.col("product").str.contains("Widget"))
3df.filter(pl.col("qty").is_between(5, 10))

Add / modify columns #

1df.with_columns(
2    total=pl.col("price") * pl.col("qty"),
3    category=pl.col("product").str.replace(r"\s.*", ""),
4)

GroupBy #

1(df.group_by("product")
2   .agg(
3       pl.sum("qty").alias("total_qty"),
4       pl.mean("price").alias("avg_price"),
5       pl.col("id").count().alias("order_count"),
6   ))

Aggregations can be multiple and named in one call — no need for separate .agg() calls.

Sort #

1df.sort("price", descending=True)
2df.sort(["product", "price"])   # multiple columns

Join #

1other = pl.DataFrame({
2    "product": ["Widget A", "Gadget B"],
3    "warehouse": ["WH-01", "WH-03"],
4})
5
6df.join(other, on="product", how="left")

Supported join types: "inner", "left", "outer", "cross", "anti", "semi", "asof".

Pivot / Melt #

1df.pivot(index="date", columns="product", values="qty", aggregate_function="sum")
2
3df.melt(id_vars=["id"], value_vars=["price", "qty"],
4        variable_name="metric", value_name="val")

Window functions (over) #

1df.with_columns(
2    running_total=pl.col("qty").cum_sum().over("product"),
3    rank=pl.col("price").rank("dense").over("product"),
4)

The Expressions API #

What makes Polars uniquely powerful is its expression system — every operation returns an expression, and expressions can be composed, named, and reused.

Expression building blocks #

 1# Column references
 2pl.col("price")                    # one column
 3pl.col("price", "qty")             # multiple columns
 4pl.col("*")                        # all columns
 5pl.col("^price|qty$")              # regex matching columns
 6
 7# Arithmetic
 8(pl.col("price") * 1.1).round(2)
 9
10# String operations
11pl.col("product").str.to_lowercase()
12pl.col("product").str.replace("Widget", "Gadget")
13pl.col("product").str.split(" ").list.first()
14
15# Temporal operations
16pl.col("date").dt.year()
17pl.col("date").dt.weekday()
18
19# Conditional logic
20pl.when(pl.col("price").is_null())
21  .then(0)
22  .otherwise(pl.col("price"))
23  .alias("price_filled")
24
25# Rolling / window
26pl.col("qty").rolling_sum(window_size=3)
27pl.col("price").shift(1)
28
29# Cumulative
30pl.col("qty").cum_sum()
31pl.col("price").cum_min()

Named expressions and reusability #

Because expressions are just objects, you can factor them out:

1revenue_expr = (pl.col("price").fill_null(0) * pl.col("qty")).alias("revenue")
2avg_expr = pl.col("qty").mean().over("product").alias("product_avg_qty")
3
4df.with_columns(revenue_expr, avg_expr)

Contexts #

Every expression is evaluated in one of three contexts:

Understanding this makes the API predictable — once you know the context, you know how the expression will be applied.

Lazy vs. Eager Execution #

Eager API #

1# All operations executed immediately
2df.filter(...).select(...).group_by(...).agg(...)

Every call materializes a result. Good for interactive exploration, but less optimal for complex pipelines.

Lazy API #

1# Build a query plan, execute at the end
2q = (df.lazy()
3       .filter(pl.col("price").is_not_null())
4       .with_columns(total=pl.col("price") * pl.col("qty"))
5       .group_by("product")
6       .agg(pl.sum("total"))
7     )
8plan = q.explain()   # view the optimized plan
9result = q.collect() # execute

What the optimizer does for you:

• Predicate pushdown — Filters are pushed as close to the data source as possible.
• Projection pushdown — Only the columns you need are read from disk.
• Join reordering — Smaller tables are broadcast or hash-joined advantageously.
• Slice pushdown — limit/offset is pushed to the data source.

Streaming mode (for datasets larger than RAM) #

1result = q.collect(streaming=True)

This processes data in chunks, spilling intermediate results to disk when necessary. Polars is one of the few Python DataFrame libraries with native streaming built in, not as an afterthought.

Performance #

Polars consistently outperforms pandas and often rivals Apache Spark on single-node workloads. Here are typical speedups on common operations:

These are illustrative — actual performance depends on hardware, data shape, and operations. Always benchmark against your own workload.

Why is Polars so fast?

1. Written in Rust — compiled to native code, no Python interpreter overhead in the hot path.
2. Multi-threaded by default — uses all available CPU cores via Rayon.
3. SIMD vectorization — via Apache Arrow's compute kernels.
4. Cache-conscious algorithms — data is laid out in Arrow columnar format, friendly to CPU caches.
5. Query optimization — the lazy planner eliminates redundant work before a single row is touched.

Interoperability with pandas #

You can seamlessly move data between Polars and pandas:

 1# Polars → pandas
 2df_pd = df.to_pandas()
 3
 4# pandas → Polars
 5df_pl = pl.from_pandas(df_pd)
 6
 7# With zero-copy when using PyArrow behind pandas
 8df_pd = pd.DataFrame({"x": [1, 2, 3]})
 9df_pd = df_pd.convert_dtypes(dtype_backend="pyarrow")
10df_pl = pl.from_pandas(df_pd)  # zero-copy

This makes Polars easy to adopt incrementally — you can start using it for performance-critical sections while keeping pandas for the rest.

Real-World Examples #

Example 1: Parse and aggregate server logs #

 1# 10 million log lines, CSV, ~2 GB
 2logs = pl.scan_csv("logs.csv")
 3
 4summary = (logs
 5    .filter(pl.col("status_code") != 200)
 6    .group_by("endpoint")
 7    .agg([
 8        pl.count("request_id").alias("failure_count"),
 9        pl.mean("response_time_ms").alias("avg_response_time"),
10        pl.col("response_time_ms").max().alias("max_response_time"),
11    ])
12    .sort("failure_count", descending=True)
13    .collect(streaming=True)
14)

Example 2: Time-series resampling #

 1df = pl.DataFrame({
 2    "timestamp": pl.date_range(
 3        "2025-01-01 00:00", "2025-01-01 23:59",
 4        interval="30s", eager=True
 5    ),
 6    "value": pl.Series([i % 100 for i in range(2880)]),
 7})
 8
 9hourly = (df
10    .group_by_dynamic(
11        index_column="timestamp",
12        every="1h",
13        closed="left",
14    )
15    .agg([
16        pl.mean("value").alias("avg"),
17        pl.min("value").alias("min"),
18        pl.max("value").alias("max"),
19    ])
20)

Example 3: Multi-table merge with complex logic #

 1sales = pl.read_parquet("sales.parquet")
 2products = pl.read_parquet("products.parquet")
 3inventory = pl.read_parquet("inventory.parquet")
 4
 5result = (sales
 6    .join(products, on="sku", how="left")
 7    .join(inventory, on="location_id", how="inner")
 8    .filter(pl.col("stock_qty") >= pl.col("order_qty"))
 9    .with_columns(
10        fulfillment=pl.when(pl.col("stock_qty") >= pl.col("order_qty") * 2)
11                     .then("direct")
12                     .otherwise("partial")
13    )
14    .group_by(["fulfillment", "category"])
15    .agg(pl.sum("order_qty").alias("units"))
16    .sort("units", descending=True)
17)

Best Practices and Tips #

1. Use scan_* for large files — pl.scan_csv() / pl.scan_parquet() builds a lazy query without loading the whole file.
2. Prefer with_columns over repeated select — fewer passes over the data.
3. Use collect(streaming=True) for out-of-core processing.
4. Name intermediate expressions — improves readability and enables reuse.
5. Use .explain() on lazy queries to verify the optimizer is applying pushdowns.
6. Avoid apply() — it drops back to Python and kills performance. Use native expressions whenever possible.
7. Leverage pl.col("*") and regex to write DRY selections.
8. Profile with df.profile() to see per-operation timing.

When to Use Polars (vs. pandas vs. Spark) #

Conclusion #

Polars is not just "a faster pandas" — it's a rethinking of how a DataFrame library should work in the modern multi-core, memory-constrained era. Its expression API is both elegant and powerful; its query optimizer removes mental overhead; and its performance speaks for itself.

Whether you're building production ETL pipelines, analyzing terabytes of log data, or just tired of waiting for pandas to finish a groupby, Polars deserves a spot in your toolkit.

Further reading #

• Official Polars documentation
• Polars User Guide
• Polars GitHub repository
• Apache Arrow
• h2oai database benchmark