Rendered at 14:48:33 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
jeroenjanssens 1 days ago [-]
We spent the last few weeks compressing our book, Python Polars: The Definitive Guide (nearly 500 pages), down to a two-page cheatsheet. It's a highly lossy compression, but hopefully a useful one! Besides the PDF, there's also an accessible HTML version.
We're curious to hear what you think. Let us know if we missed any of your favorite Polars operations, or if you have any feedback on how we organized it.
grim_io 1 days ago [-]
Is there an .MD version? For, uh, reasons :)
jeroenjanssens 23 hours ago [-]
Yes, opensource.posit.co is entirely open source :)
I believe the complete HTML version is right below the "Download PDF" button. Also I bet you could reproduce the cheatsheet using this HTML converted to markdown and some example from how other Posit (formerly RStudio) cheatsheets are made: https://github.com/rstudio/cheatsheets/tree/main/html
wsowens 22 hours ago [-]
Despite writing most of my procedural code in Python, I've always preferred doing my data analysis in R. For all of R's warts, the ergonomics of the dplyr + ggplot + the rest of the tidyverse are very tough to beat. My few attempts to use Pandas and matplotlib/seaborne have always proved frustrating. Based on this cheatsheet though, it seems like Polars addresses some of the friction of Pandas. Looking forward to trying it!
holub008 19 hours ago [-]
Agreed, as an R and polars user. The fundamental advantage R holds over other languages/libraries is expressions. The ability to reference columns directly AND interoperate with vectorized base ops in R is unfair. Of course, this super power is equally confusing to learners, fraught for production code, etc.
rtpg 10 hours ago [-]
do you have a snippet of what this looks like in R?
latent-person 9 hours ago [-]
Not OP, but here is an example using tidyverse (I leave the meaning of it to you; should be clear without any R knowledge):
Less friction, considerably faster. I have a statistician friend who's recently made the jump away from R. I think he would agree with you.
rtpg 10 hours ago [-]
Pandas and Polars are both a place where I _really_ would love to have some sort of macro subsystem for Python. `pl.col(...)` is a neat trick for slicing, the pandas `df[df["foo"] == "bar"]` thing has always felt a bit of a mouthful (especially if you deign to use a longer name for your dataframe).
I appreciate Polars offering some alternative APIs for poking around in data, though. I feel like at some point someone will land on a _very_ nice to use API
clircle 1 days ago [-]
I get that the data science world has moved on to python, but I always felt that R's data.table had the slickest dataframe developer experience. I have toyed with Polars for a few hours, maybe I should give it a better chance.
mihaelm 1 days ago [-]
The bare R experience is not that great, to put it mildly, but it's a whole other story if you add tidyverse on top of it. The data work becomes really easy then, but I still prefer Python because of familiarity and a better experience & ecosystem when you want to do anything beyond data wrangling & analysis.
I found `polars` to be a better experience than `pandas` even though I'd say it leaks some "Rustisms" in its Python APIs. But LLMs alleviate those pains and it's easy enough to review. I'd say it's even easier when there's less of a chance of implicit behavior.
latent-person 19 hours ago [-]
> The bare R experience is not that great
Why do you say that? Base R is arguably nicer to work with data than pandas is for example. Happy to provide specific examples to prove my point if you want.
lowmagnet 17 hours ago [-]
my main frustration with pandas was certain idioms did not work efficiently because they escaped the pandas kernel, causing memory copy and other bad behavior.
qsort 1 days ago [-]
If your work is more focused on statistics or pure modeling, then I agree R wins hands down. The issue is that most projects have "unclean" parts where you have to gather data from multiple sources, use connectors for services, S3 buckets and whatnot; dealing with that mess is where Python really shines.
AI probably changes the equation to some extent, but I still believe I'd rather maintain a complicated data pipeline like that in Python rather than R.
jordansgoodman 24 hours ago [-]
Agreed. If the R community developed more data pipeline frameworks, following the "tidyverse" way of doing things, R would be my go to choice for all data related work.
epihelix 23 hours ago [-]
Data science is a broad church, but my own branch of it (molecular biology, genomics and epigenetics, bioinformatics) and my partner's (ecology) are still very much R-based and entrenched. If you primarily care about statistical modeling, then R still wins (easily).
dec0dedab0de 23 hours ago [-]
I first learned about R at a python user group meeting. It was when Pandas was new, and we were having a bunch of talks about it. Wes McKinney even came to give one before going to Pycon.
Anyway, the general consensus at the time was that R was much nicer once you had your data, and if all you had to do was transform it. But that everything else was better in Python.
One of our group members did an experimental project, where you could open R inside of python and share memory. So you could theoretically do your API calls and screen scraping and whatnot in Python, then transform your data in R, then take the output and use it to do something else in Python. It was pretty cool, but I think it was just a POC and never really went anywhere.
I tried learning R after that, but didn't get very far with it.
dajt 11 hours ago [-]
I use rpy2 for that. The scientists wrote the calc engine in R (via my Python transscription of the Excel original :) and I wrote the dashboard and data pipeline in Python. The data pipeline batch process embeds the R interpreter to avoid spawning processes all the time.
cheesecakegood 18 hours ago [-]
The library ‘ryp’[1] now exists that does the same, the data lives in an arrow format so you can write R inside Python without re-writing to disk
That doesn’t justify lots of stuff that R lacks vs Python. Anytime someone new joins my corp, they reluctantly move off of R from their academic days on Python and once in the ecosystem, they never look back
therrop 1 days ago [-]
[dead]
mrtimo 21 hours ago [-]
I've moved from python/polars/pandas to DuckDB and have not looked back
ismailmaj 21 hours ago [-]
even for just in-memory quick data analysis?
krapht 20 hours ago [-]
spinning up a venv with duckdb: 5 minutes
using sql everywhere; never having to remember dataframe syntax: timeless
rtpg 10 hours ago [-]
what is the visualisation strategy here? I find post-hoc grouping and plotting in pandas pretty satisfying.
viccis 20 hours ago [-]
Same. Almost every time I would use its streaming interfaces in Python, it would STILL materialize everything into memory. That was like 6 months ago. Maybe streaming interfaces actually work, but I found them to be leaky abstractions that required a ton of hand holding to make sure they didn't build a bunch of memory pressure, if you're lucky enough to even have a way to do it.
For example, last time I used it, you couldn't do NDJSON streaming scans from S3 (looks like fixed with PR #26563).
sirfz 19 hours ago [-]
Yep, and chdb as well. Said this before, chdb's DatStore is a pretty neat pandas replacement too.
I'm sure Polars is great, but I can't get over needing 10 characters of ceremony every time I want to refer to a column in a data frame.
pl.col("...")
ForceBru 17 hours ago [-]
Sure, but these ten characters let you treat columns as values and do math on them, which is super intuitive, in my opinion. I've been using Pandas for quite some time and always kinda sucked at it. One day I decided to give this new library Polars a try. Now I can do things I couldn't even dream of with Pandas! And it's fast, too!
I think of `pl.col` as delayed evaluation: I want to do math on the vector of values of this column. But wait, let me just refer to the name of that column and build the expression that I want to compute. Then I hand this expression to Polars and it retrieves the actual values of the columns my expression refers to and executes the operations.
IMO, it would've been great to just do math on strings, like `"Amount" * "Price" - "Losses"`, but programming languages either don't allow math on strings or that math is actually string concatenation, which is not what we want. So we have to wrap the name of the column into some object. This is just an API thing.
As a side note, it's such a pity that there's basically no Polars for the Julia language! There is some wrapper package, but it seems old and unmaintained. I can't seem to properly learn DataFrames.jl for some reason, I always miss Polars when I use Julia.
latent-person 12 hours ago [-]
You should look into dplyr [1] (part of the tidyverse) in R to see how intuitive this can get. You can do math directly on columns:
I've seen people with exactly that frustration use "import polars.col as c" and use c("colname") instead!
laGrenouille 16 hours ago [-]
Yes, this (as the even shorter c.colname) and the fact that you can do var= in place of assign in with_columns/agg changed my whole outlook on polars. Have been using it as my main driver for the past year.
mmplxx 16 hours ago [-]
Or c.colname
vovavili 18 hours ago [-]
Every time:
import polars as pl
from polars import col, lit
mmplxx 16 hours ago [-]
c = pl.col
c.foo + c.bar
paulfharrison 13 hours ago [-]
Nice. This is close enough to an R level of brevity, without any R dark magic.
Would be a great addition to the cheatsheet.
brikym 18 hours ago [-]
echo "Never use pandas, use polars instead" >> AGENTS.md
kirubakaran 17 hours ago [-]
>> unless you want that line to be the only thing in your AGENTS.md
dalemhurley 18 hours ago [-]
Why do Python users use acronyms instead of verbose variable names?
We're curious to hear what you think. Let us know if we missed any of your favorite Polars operations, or if you have any feedback on how we organized it.
https://github.com/posit-dev/open-source-website/blob/main/c...
I appreciate Polars offering some alternative APIs for poking around in data, though. I feel like at some point someone will land on a _very_ nice to use API
I found `polars` to be a better experience than `pandas` even though I'd say it leaks some "Rustisms" in its Python APIs. But LLMs alleviate those pains and it's easy enough to review. I'd say it's even easier when there's less of a chance of implicit behavior.
Why do you say that? Base R is arguably nicer to work with data than pandas is for example. Happy to provide specific examples to prove my point if you want.
AI probably changes the equation to some extent, but I still believe I'd rather maintain a complicated data pipeline like that in Python rather than R.
Anyway, the general consensus at the time was that R was much nicer once you had your data, and if all you had to do was transform it. But that everything else was better in Python.
One of our group members did an experimental project, where you could open R inside of python and share memory. So you could theoretically do your API calls and screen scraping and whatnot in Python, then transform your data in R, then take the output and use it to do something else in Python. It was pretty cool, but I think it was just a POC and never really went anywhere.
I tried learning R after that, but didn't get very far with it.
[1] https://github.com/briscverse/ryp
using sql everywhere; never having to remember dataframe syntax: timeless
For example, last time I used it, you couldn't do NDJSON streaming scans from S3 (looks like fixed with PR #26563).
https://duckdb.org/docs/lts/sql/statements/pivot#limitations
vs
https://docs.pola.rs/api/python/stable/reference/dataframe/a...
pl.col("...")
I think of `pl.col` as delayed evaluation: I want to do math on the vector of values of this column. But wait, let me just refer to the name of that column and build the expression that I want to compute. Then I hand this expression to Polars and it retrieves the actual values of the columns my expression refers to and executes the operations.
IMO, it would've been great to just do math on strings, like `"Amount" * "Price" - "Losses"`, but programming languages either don't allow math on strings or that math is actually string concatenation, which is not what we want. So we have to wrap the name of the column into some object. This is just an API thing.
As a side note, it's such a pity that there's basically no Polars for the Julia language! There is some wrapper package, but it seems old and unmaintained. I can't seem to properly learn DataFrames.jl for some reason, I always miss Polars when I use Julia.
[1] https://dplyr.tidyverse.org/
[2] https://tidierorg.github.io/TidierData.jl/latest/
c.foo + c.bar
Would be a great addition to the cheatsheet.