Pandas Should Go Extinct(eddie.codes) |
Pandas Should Go Extinct(eddie.codes) |
In many cases I’ve found directly using python primitives to be less confusing than pandas.
Similarly, in companies I’ve worked at, the datasets just aren’t that big. Especially if you’ve got access to modern hardware.
I'm shocked! Shocked, I say!
Sometimes you just want something that works with all the things. That's pandas. But, like the bamboo eaters, it wont be long...
For dumb simple select group by OLAP queries on medium data ? Sure clickhouse local or duckdb works perfectly well.
But if you need to construct dataset ? Or process existing dataset, do heavy filtering, transformation, reshaping, splitting? The proper ETL work, then pandas is really the perfect use case.
And pandas can work with small memory footprint as well, its actually trivial to do that, plus there are libraries like Modin that are upgrades over pandas with pandas api
Pandas is the swiss knife tool of data science that lets you do anything with the data and it integrates well with ML libraries
Also, pandas's group_by() and window function pipelines made me want to tear my hair out while both are easy in polars.
Use SQL if you need to move data around.
And in the end save processed data in aws s3 in another format like iceberg or whatever
In pandas a lot of these are one-liners that are impossible in sql (depends on what sql engine you have?)
I'd write more tutorials about how to use data science tooling but one consequence of AI is that all the old data sources I used to analyze such as social media and Reddit are now completely locked down (I am surprised NYC Taxi is still being updated, though). Therefore in the meantime, I'm working on making better data science tooling...although unclear to what end due to the data issue above.
I see a lot of work towards Geo-Data Science.
PostGIS was a first generation. Now we have GeoPandas, DuckDB Geoanalysis tools and Sedona DB.
IMDB data set is also up.
Instead it's a cute bait-and-switch title, and the article tells you upfront that it's actually about the Python `pandas` library. Which I think I've encountered maybe once in my entire career (I'm not in the data-science field), so I don't have much meaningful to say about the article itself. I just want to commend the author on fooling me with the title. This is the kind of "clickbait" I can respect and actually wish there was a little bit more of sometimes. A nice chuckle, then a real article.
[1] https://www.goodreads.com/book/show/9656950-do-we-need-panda... [2] https://www.theguardian.com/environment/2009/sep/23/panda-ex...
It is impossible, and has been tried many times by many people, to reproduce Octopus vulgari in captivity. Their reproductive behaviour looks also quite weird for us (the mother always die). Yet they thrive in the wild unless over fished.
Would be very interested to read an article on the literal interpretation of my title.
Also just wanted to say thank you for this comment - made my day :)
Assumes that a project moves beyond 10s of GBs. I guess 99.9% of projects that import pandas fall well below this threshold.
This isn't a call to arms to rewrite everything in the new shiny, just consider the new shiny for new shiny things
Credit to pandas for popularizing dataframes in Python, but polars and duckdb are objectively better APIs in addition to their implementation improvements. Agree that it’s time we let it go.
If a library is performant at large datasets, it is likely performant at small ones too.
Don't skip your profiling
Regardless, CPUs are really good at single-thread.
I agree that the API has warts, though typically it is more concise than polars.
[1]https://bjarkehautop.github.io/Website/blog/data-wrangling-t...
https://eddie.codes/posts/pandas-should-go-extinct/ <=> https://eddie.codes/posts/source-code-comments/
Published two posts at the same time and total PEBCAK
Pandas is not perfect, it has a bunch of warts. But it is good enough for most. (And many of those folks are using Excel or tableau or power bi... These were the types I was training this week).
If you have medium data, migrating from pyarrow backed pandas to duck or Polars is trivial.
If I still need to do db-like things in python I think duckdb is better.
> TODO: rewrite this entire service
remains.
I shall pass this information along to my sleep paralysis demons. They'll be glad to hear it.
> > TODO: rewrite this entire service
Part of me enjoys the historical significance of this. However, I feel like keeping it would violate my own princples: https://eddie.codes/posts/source-code-comments/
I still prefer (and use) Pandas for EDA. I think matplotlib integration is a better choice for most viz.
Also, I'm probably in the top 3-5 worldwide for number of folks I've trained with pandas. I offer Polars training and there is little demand for it.
DuckDB on the other hand is wonderful and truly doesn't use any more memory than it really needs to.
Thijs, DevRel @ Polars
I reported many bugs to both pandas and polars over the years and both teams tend to address relatively big ones. (Still have some outstanding pandas bugs that I think are a big deal but the devs disagree.)
I think we can be certain that reduced cash use didn't cause the pandemic, and a common cause for both is also hard to imagine. That only leaves the pandemic causing a reduction in cash use. Am I missing something?
The problem is, orgs who see themselves as big data orgs want to act that way, even if they're medium data. "But we'll need it when we grow", "we need to know the state of the art tools"
While throwing hardware at problems isn't always a reasonable solution, it's trivialized to discuss this without IOPS or other read limitations.. and the nvme enterprise market is rapidly developing making this change quickly.
Something going wonky on their blog, where two posts got their links swapped.
This and the decision not to natively plot with matplotlib keep me in pandas for most tasks. (Plus there's is still a relatively large demand for pandas training.)
I do not understand the aggressiveness towards previous evolutionary steps. Who knows, may they come up with newer methods. I still use it. I also use DuckDB and Polars. DuckDB is more useful to me these days from either. Still Pandas has a place in my arsenal. Mostly because I know it better and I like it.
There is also libraries for doing type checking.
The i5-1135G7 has only 4 cores. Hyperthreading is enabled, so it reports 8 "virtual processors".
Hoping OP can fix this on their end so the url has the expected content. Whoops!
Point is, if you're miserable in the place where you are, consider whether moving to a different town is feasible for you. It's not always an option for many people, for many varied reasons. But it's definitely something to consider. If it turns out you can do it, sometimes taking that risk can pay off in the long run.
Lots of folks complain about it before using it. After they use it it is a non issue.
If it really is an issue (and it generally isn't a cause of vectorization removal when used correctly) and you can't get over the syntactic noise if the lambda, pandas 3 introduced pd.col (that work in most (I filed a big about some exceptions) places when you'd use lambda).
edit:
Let me clarify. From the blog-post:
> since a `DataFrameGroupBy` object doesn’t have a `.query()` or boolean-indexing shortcut of its own, so filtering within groups needs `.apply()` again, and the surrounding pipeline has to be rebuilt around it:
Hence you really do need one of the versions of the code I gave. You can't do the naive approach with just `.groupby().filter(lambda: )`, since you need a row-wise decision.
(sales
.assign(country_median=lambda df_: (
df_.groupby("country")["amount"].transform("median")
))
.query("amount <= country_median * 10")
.assign(net=pd.col('amount') - pd.col('discount'))
.groupby("country", as_index=False)
.agg(total=("net", "sum"))
)It's late here, I'm going to bed, perhaps I'll write the code tomorrow when I'm at my laptop and not on my phone.