Pre-Release of Polars 2.0(pola.rs) |
Pre-Release of Polars 2.0(pola.rs) |
I know this take reveals me as a very dull person, but I love seeing projects take semver seriously like this! Version bumps should really be about removing deprecated cruft rather than shiny new features.
I've used polars for a while now, and their focus on stability was a big part if convincing me to make the jump initially!
That's how I thought semantic versioning worked
I don't know how to read this sentence other than "there are breaking changes we want to make"
Can there be deprecated cruft without new features? :-D
All new shiny new features shouldn't have waited for the (N+1).0 version, they should already have been part of the (N).(M) version.
In practice, the removing the deprecated cruft will remove blockers for some new features, but that should be rare.
Pandas tends to push all problems to runtime, with all sorts of hidden heuristics. Particularly around column types and missing values. It's very hard to know if you've tested all the edge cases. The only way to test your code is to throw all variations of data at it. Fine if you're sitting at a notebook and have the patience to validate and "clean" the data on its behalf. Not so fine if you get paged at 3am because your data pipeline failed when it expected an int column but got float.
Polars is more strict by default and front-loads costs through its planner. The resulting apps are noticeably more stable in production. You can test code and reasonable assurance that it will work on data in the wild.
I don't really have any interest in the API ergonomics or syntax - both are fine. It's all about how they deal with data variation at runtime. Can you write general code that doesn't break on variants? Pandas, not a chance. Polars, absolutely!
Bonus round: polars has a Rust API too, the compiler can effectively prove that your program handles every edge case. It's common to write rust polars apps that run unattended for years.
That's why polars is a great option. Start prototyping in python, then a relatively easy port to a Rust app when the need hits.
We recently added a Polars backend to GFQL (cypher graph queries on dataframes, no DB needed), both CPU and GPU mode, and super impressive. Noticeable improvements vs pandas/cudf, and enabled GFQL to beat out popular systems on more categories like low-latency, not just big datasets: https://www.graphistry.com/blog/cypher-on-polars-cpu-gpu-gra...
But they were a bit quiet lately, and I started looking more and more into DuckDB recently… until the recent acquisition of DuckLab by AWS
After using pandas for 10 years, I favor SQL now, for some reason.
I use SQL in data pipelines and processing that is going to require interoperability.
But for data exploration, I usually prefer Polars (imo it is easier to work with text, semi-structured data, etc.)
>pl.Series([None, 1, 0, 2], dtype=pl.UInt32).cast(pl.Enum(["a", "b", "c"])) ># ComputeError: casting from u32 to enum is not supported.
>Use instead: .cat.to(dtype) for int → categorical, .cat.physical() for >categorical → int.
Always show the correct way of doing things. I have no idea what the correct is here, and I don't really see what benefit this change in API brings.
You can find it in the migration guide. Let me know if you miss anything, if you'd like you can make an issue and I'll make sure to get to it before the 2.0 release!
So strange that it's now relatively normal to see a typo and think 'oh cool, a human wrote this, I can take this seriously' rather than 'oh dear, they can't spell, I can't take this seriously'.
The name was chosen early on to contrast with the old execution model, which was essentially all-data-in-memory, column-at-a-time. That engine still exists, we use it as a fallback mechanism for things that aren't supported yet in the new engine (or if you explicitly ask for `engine="in-memory"`).
The new execution model first constructs a computational graph of nodes which communicate in streams of in-cache batches (morsels) of data, meaning the full dataset will never be held in memory if not necessary. This was called the streaming engine for that reason in an early prototype and the name stuck. In hindsight I do admit the naming choice is somewhat confusing.
(Or a more general question: What is the best resource for me to read about how the streaming engine and cache work?)
I'm not sure if I agree that "hidden setting actually keeps your data correct" is something that should be the default.
[1] Aha, now I see why language models use this so frequently and why it might be overrepresented in the data. This is a perfect way to move the blame from the person you're responding to, if they're mistaken. They probably have a super, super overtuned "politeness" gym using sentiment analysis that tries to reword answers to not blame the misunderstandings of the person. Then this blame shifting unfortunately gets re-used as this super, super common phrase.
> Using algorithms that don't need to upkeep the ordinality requirement in every operation will definitely move the library to a better direction and make future data modeling better and more explicit.
How would the library "make future data modeling ... more explicit" if this is a change to a default, which is implicit?
They do not guarantee stability of hashing algorithm or seed across versions and platforms.
(No complaints here! I agree ordering shouldn’t matter unless you make it explicit.)
I do not think of a dataframe as a set, but an ordered collection of rows. My source csv had the rows in this order and I want that maintained unless I choose maximum performance.
- much faster, multithreaded by default. Read in a big csv with it and see how it feels.
- no index/MultiIndex. Pandas special treatment of index always felt like more trouble than it was worth, so no need to reset_index() everywhere.
- expressions are very portable. At first using pl.col everywhere feels like a bit much, but you can define them anywhere and then apply them to a dataframe whenever you want.
- once internalized, the syntax makes much more sense and is far more consistent compared to pandas.
Of course all depends on what your use cases are. If performance is important then I'd strongly recommend trying it out. If you just use it to have a look at the odd dataframe, maybe not worth your time as much
Pandas is more ergonomic in that some ideas can be more tersely represented. The downside is that this results in more dynamism which can change if the underlying data gets updated. Polars is more strict in that it will not silently flip a data type on you. However this strictness does come at the cost of being a bit slower to type and some data idioms not having a good Polars equivalent.
People like to note the speed improvements, but that is the least interesting thing about the library. Rarely have I ever had a problem where I was bottlenecked by Pandas throughout.
Polars is very much a Pandas 2.0 with a bunch of lessons learned. I do not think it is earth shattering changes, but it is worth migrating when you can.
This is not a criticism. As someone who doesn’t use Python, I simply found it amusing.
Pandas is a mess though.
df.select(
pl.col("x"),
(pl.col("w")/pl.col("z")).alias("y")
with
df |> select(x, y = w/z)
The duckdb python api is okay, but it is a bit limited, no ctes, no as of join, and it can be slow at bind/interpretation time when you do stuff like unioning multiple relations in a loop (I think that becomes O(N^2), but I might be wrong). Most issues can be worked around, but Polars is designed from the ground up to be used from python.
Unfortunately, polars does not support parameterized queries, so the risk of SQL injection is extremely high.
https://docs.pola.rs/development/versioning/
> Polars adheres to the semantic versioning specification:
And it does have breaking changes in 2.0. The original asker presumably missed that.
E:
On the other hand, that whole page on versioning seems inconsistent.
Regular CH also support external data sources, so I can read 500GB of JSON from S3 and group by it on production server very fast and in memory.
Requiring code to be explicit about its invariants by default is just plain good design and makes things more robust. And when being more exact about invariants, you can reap the algorithmic benefits.
It would become more explicit because where the order matters, you will now see a `maintain_order=True` where previously, you couldn't say whether it actually needed the fixed order or not.
Neither R nor Pandas will shuffle results by default.
maintain_order=False
?You seem to suggest they did it for benchmarking reasons only. They could use the option there themselves without changing the default so that is unlikely to be the motivation.
I think on basic queries, SQL is really nice, but when stuff gets more complex, with a bunch of CTEs, let alone functions requiring loops, it becomes pretty obtuse.
But no, that's not what I meant. I meant that the batch is meant to be of a size that fits in your CPU cache. This can be a huge throughput improvement as each bit of data stays in cache as it moves from data source to sink.
Compare this to column-at-a-time execution: by the time you start the next operation on this column the start of the column will be out of cache again, meaning you operate at RAM speed (or worse, disk speed) rather than cache speed.
I gave a (fairly surface-level) talk on the streaming engine a bit over a year ago: https://pola.rs/posts/talk-polars-meetup-1-streaming-engine/.
Python's linters leave many holes unfilled - it catches some stuff but still leaves everything else to the runtime. Literally every Python application that I've worked on in the last 3 years a) is 100% type checked yet b) still hits massive numbers of novel runtime errors on the regular. Python's anemic type system is in no way a substitute for an advanced compiler.
At some point you reach the Hyrum's Law / Spacebar Heating edge where sure, there's no reasonable technical definition which concludes your change "broke" people but somebody's solution used to work and now it doesn't and they're angry about that.
To give an example that's only slightly less ridiculous than spacebar heating, maybe somebody depended on being able to parse a file from your library, you re-designed the library, there are no API changes but they're parsing your source code and now their approach doesn't work.
Generally, libraries with fewer and lighter users tend to be more lax, maybe they forgot to make a structure opaque, that's a break but none of their users notice, or they forgot to make an error type non-exhaustive, but everybody using their type had a default clause so no harm done when they add a new value.
Hyrum means that as you get bigger and more important, some day, the spacebar heating problem happens and too bad. A language (like Rust) can help you forestall that day somewhat, and reduce the blast radius when it does, but it is inevitable, Hyrum's law wasn't something you can outright prevent from happening, it's a factor for engineers to consider when designing systems, like misuse resistance.
I find that SQL is only easier to read with minimal abstraction, but as soon as the project gets bigger SQL becomes an unwieldy island of different that has served its purpose after we’re done with reading/writing the data.
import polars as pl
# 1. Base Dataset
lazy_df = pl.LazyFrame(
{
"store_id": ["S01", "S02", "S03", "S04", "S05"],
"revenue": [5000.0, 2400.0, 15000.0, 900.0, 3200.0],
"margin": [0.45, 0.30, 0.60, 0.15, 0.50],
"tx_count": [120, 45, 300, 20, 85],
"returns": [5, 12, 45, 2, 8],
}
)
# 2. Define Layer Abstractions
def get_kpi_layer() -> list[pl.Expr]:
return [
(pl.col("returns") / pl.col("tx_count")).alias("return_rate"),
(pl.col("revenue") / pl.col("tx_count")).alias("avg_order_value"),
]
def get_threshold_layer(thresholds: dict[str, list[float]]) -> list[pl.Expr]:
return [
(pl.col(col) > limit).alias(f"is_{col}above{int(limit)}")
for col, limits in thresholds.items()
for limit in limits
]
def get_interaction_layer(numeric_cols: list[str]) -> list[pl.Expr]:
return [
(pl.col(a) / (pl.col(b) + 1e-5)).alias(f"ratio_{a}per{b}")
for i, a in enumerate(numeric_cols)
for b in numeric_cols[i + 1 :]
]
def get_segmentation_layer() -> list[pl.Expr]:
return [
pl.when(pl.col("margin") > 0.4)
.then(pl.literal("High"))
.otherwise(pl.literal("Low"))
.alias("margin_profile")
]
# 3. Consolidate and Execute Single Graph Pass
thresholds = {"revenue": [1000.0, 5000.0, 10000.0], "tx_count": [50, 100, 200]}
numeric_cols = ["revenue", "margin", "tx_count", "returns"]
expr_pool = [
*get_kpi_layer(),
*get_threshold_layer(thresholds),
*get_interaction_layer(numeric_cols),
*get_segmentation_layer(),
]
final_df = lazy_df.with_columns(expr_pool).collect()> I do have type and dirty data problems of the dataframes/csvs/parquet I'm reading.
Something about those two statements makes my brain explode. You rarely have type problems, except when dealing with data? Isn't the whole point of a program to deal with data?
from polars import col as C
df.select(C.x, y = C.w / C.z)ggplot vs matplotlib
dplyr vs pandas
And I loved that everything in RStudio was so easily inspectable. Have a huge dataframe? Just look at it right in your IDE.
`df.select("x", y=pl.col.w/pl.col.z)`
WITH raw_data AS (
SELECT * FROM (
VALUES
('S01', 5000.0, 0.45, 120, 5),
('S02', 2400.0, 0.30, 45, 12),
('S03', 15000.0, 0.60, 300, 45),
('S04', 900.0, 0.15, 20, 2),
('S05', 3200.0, 0.50, 85, 8)
) AS t(store_id, revenue, margin, tx_count, returns)),
base_data AS (
SELECT
store_id,
revenue,
margin,
CAST(tx_count AS DOUBLE) AS tx_count,
CAST(returns AS DOUBLE) AS returns
FROM raw_data
)
SELECT
store_id,
revenue,
margin,
CAST(tx_count AS BIGINT) AS tx_count,
CAST(returns AS BIGINT) AS returns,
-- KPI Layer
returns / tx_count AS return_rate,
revenue / tx_count AS avg_order_value,
-- Threshold Layer (matching original alias names)
revenue > 1000.0 AS is_revenueabove1000,
revenue > 5000.0 AS is_revenueabove5000,
revenue > 10000.0 AS is_revenueabove10000,
tx_count > 50 AS is_tx_countabove50,
tx_count > 100 AS is_tx_countabove100,
tx_count > 200 AS is_tx_countabove200,
-- Interaction Layer (preserving exact numeric formula & aliases)
revenue / (margin + 1e-5) AS ratio_revenuepermargin,
revenue / (tx_count + 1e-5) AS ratio_revenuepertx_count,
revenue / (returns + 1e-5) AS ratio_revenueperreturns,
margin / (tx_count + 1e-5) AS ratio_marginpertx_count,
margin / (returns + 1e-5) AS ratio_marginperreturns,
tx_count / (returns + 1e-5) AS ratio_tx_countperreturns,
-- Segmentation Layer
CASE WHEN margin > 0.4 THEN 'High' ELSE 'Low' END AS margin_profile
FROM base_data;And now write it such that all the conditions and transformations are injected into the string (somehow) rather than written in explicitly. Much worse.
polars is code and can be version controlled too. Dataframes in my opinion are more elegant, and with the right backends and some lineage enhancements, could serve a much wider set of use cases than what DBT does