Polars / Chapter 5 of 7

Lazy queries and performance

Optimize Polars lazy queries with explain, streaming, and native expressions. Understand memory use and when Python UDFs add overhead.

3 topicsUpdated 22 Oct 2026

36 / 50

Query optimization and execution plans#

Read a plan to see what Polars will actually do, not only what you wrote.

An execution plan lists operations and their dependencies. Predicate pushdown moves filtering toward the source. Projection pushdown reads or passes onward only needed columns. Slice pushdown can move a limit earlier where valid. These are optimizer transformations, not guarantees that every file byte is skipped. A logical plan describes operations; a physical plan describes engine execution.

PythonCompare plans without depending on their exact text format

from pathlib import Path
from tempfile import TemporaryDirectory
import polars as pl

with TemporaryDirectory() as folder:
    path = Path(folder) / "sales.parquet"
    pl.DataFrame(
        {"region": ["Dhaka", "Sylhet"], "cents": [100, 200], "unused": ["x", "y"]}
    ).write_parquet(path)
    query = (
        pl.scan_parquet(path).filter(pl.col("cents") >= 150).select("region", "cents")
    )
    raw = query.explain(optimized=False)
    optimized = query.explain(optimized=True)
    print("plans differ:", raw != optimized)
    print("result:", query.collect().to_dicts())
    # For interactive diagnosis, uncomment:
    # print(raw)
    # print(optimized)
    # query.show_graph(plan_stage="physical", engine="streaming")

TextExpected output

plans differ: True
result: [{'region': 'Sylhet', 'cents': 200}]

Reading the actual plan

Uncomment explain printing to inspect the scan, selected column count, and filter placement on your version. File paths and plan wording vary, so they are not stable test output. show_graph needs Graphviz for rendered diagrams. profile executes the query and returns timing information; use it on representative data after measuring total time and memory. UDFs or collecting intermediate data can limit optimization.

Official reference: Query optimization and execution plans.

37 / 50

Streaming, memory use, and performance#

Streaming processes batches; the final result and some operation state can still grow large.

Streaming processes batches. collect(engine="streaming") still returns the full result as a DataFrame; use sink_parquet to write without collecting that result in Python. Memory use also depends on operator state.

Joins can hold lookup tables, high-cardinality groups need substantial state, and sorts may buffer data. Inspect the execution plan and measure memory on representative files; lazy execution alone does not bound memory.

PythonCompare streaming and normal results, then stream directly to a file

from pathlib import Path
from tempfile import TemporaryDirectory
import polars as pl
from polars.testing import assert_frame_equal

with TemporaryDirectory() as folder:
    root = Path(folder)
    for part in range(3):
        pl.DataFrame(
            {"region": ["Dhaka", "Sylhet"], "cents": [100 + part, 200 + part]}
        ).write_parquet(root / f"part-{part}.parquet")
    query = (
        pl.scan_parquet(str(root / "part-*.parquet"))
        .group_by("region")
        .agg(pl.col("cents").sum())
        .sort("region")
    )
    normal = query.collect(engine="in-memory")
    streaming = query.collect(engine="streaming")
    assert_frame_equal(normal, streaming)
    target = root / "report.parquet"
    query.sink_parquet(target, engine="streaming")
    assert_frame_equal(normal, pl.read_parquet(target))
    print("all execution paths:", normal.to_dicts())

TextExpected output

all execution paths: [{'region': 'Dhaka', 'cents': 303}, {'region': 'Sylhet', 'cents': 603}]

Choice	Practical effect
Scan before collecting	Allows optimization over file reads.
Select early; filter before expensive work	Reduces downstream columns/rows when semantics permit.
Native expressions instead of row loops	Avoids per-cell Python calls.
Typed IDs and sensible numeric widths	Avoids accidental casts and can reduce storage; check ranges first.
Fewer repeated collect calls	Avoids rereading and recomputing unchanged sources.
`estimated_size("mb")`	Estimates a materialized frame’s buffers, not total peak process memory.
`rechunk()`	Consolidates fragmented buffers; it may allocate/copy, so use after measuring.
Avoid unnecessary order constraints	Sorting and order preservation can constrain execution.

Memory measurements

Measure peak resident memory, output size, key cardinality, and temporary disk use as input grows. Polars runs on one machine in this workflow; storing input in the cloud does not distribute the computation.

Official reference: Streaming, memory use, and performance.

38 / 50

Python functions and UDF costs#

Use native expressions first; cross into Python for logic that native operations cannot express.

A user-defined function (UDF) runs your custom code during a query. map_elements calls Python for individual values and can be much slower than native operations. map_batches passes a Series to a function so a numerical or external library can work on a batch. return_dtype declares the result type. A function returning an expression is not a per-row UDF.

UDFs must be pure: the same input gives the same output without side effects. Polars may call them during inference or partition work differently. Do not send requests, write files, or depend on call order inside a UDF. skip_nulls determines whether map_elements calls the function on nulls. is_elementwise=True on map_batches is valid only when each output depends on its corresponding input independently, with the same row count.

PythonUse a domain parser only for a syntax native expressions do not handle here

import ipaddress
import polars as pl


def normalize_ip(value):
    try:
        return str(ipaddress.ip_address(value))
    except ValueError:
        return None


df = pl.DataFrame({"ip": ["2001:0db8::1", "192.0.2.1", "invalid", None]})
result = df.with_columns(
    pl.col("ip")
    .map_elements(normalize_ip, return_dtype=pl.String, skip_nulls=True)
    .alias("canonical")
)
print(result.to_dicts())
# Native operations for the simple surrounding cleanup:
print("nonempty source rows:", df.filter(pl.col("ip").is_not_null()).height)

TextExpected output

[{'ip': '2001:0db8::1', 'canonical': '2001:db8::1'}, {'ip': '192.0.2.1', 'canonical': '192.0.2.1'}, {'ip': 'invalid', 'canonical': None}, {'ip': None, 'canonical': None}]
nonempty source rows: 3

PythonCall a batch numerical library with an honest elementwise contract

import numpy as np
import polars as pl

df = pl.DataFrame({"amount": [1.0, 4.0, 9.0]})
result = (
    df.lazy()
    .select(
        pl.col("amount")
        .map_batches(
            lambda s: pl.Series(np.sqrt(s.to_numpy())),
            return_dtype=pl.Float64,
            is_elementwise=True,
        )
        .alias("sqrt")
    )
    .collect(engine="streaming")
)
print(result["sqrt"].to_list())
print("native equivalent:", df.select(pl.col("amount").sqrt())["amount"].to_list())

TextExpected output

[1.0, 2.0, 3.0]
native equivalent: [1.0, 2.0, 3.0]

Elementwise functions

Set is_elementwise=True only when each output depends on the corresponding input. Subtracting a whole-column mean needs it disabled. Prefer native sqrt over the NumPy callback shown here. Compiled expression plugins can avoid Python overhead for custom kernels.

Official reference: Python functions and UDF costs.

Query optimization and execution plans#

Streaming, memory use, and performance#

Python functions and UDF costs#

Help improve CodeRef