1000x faster data manipulation: vectorizing with Pandas and Numpy
Why Stack Overflow recommends it
Stack Overflow users link it when answering or asking how to replace apply, loops and iloc in Pandas with vectorized operations like np.where and np.select.
Recommended by 4 people across 4 different months, 2021 to 2024.
What it is
A PyGotham 2019 conference talk by Nathan Cheever on replacing slow row-wise Pandas operations with vectorized Pandas and NumPy code. It covers if-else logic, dictionary lookups, date calculations and string manipulation, claiming speedups of up to 1000x without Cython or extra libraries.
What people said
Recommends Tristan Salord · 2021-09 · Stack Overflow · on “np.where in pandas, checking for empty lists”
It sounds like a post digging, but i would prefer use np.where because of vectorization than list comprehension (too time costy) or apply. A lot of online tutorial deeply explain the mechanism like here [this video] .
Recommends Rycliff · 2023-01 · Stack Overflow · on “How to do a if then statement in conjunction with a group by sum operator in Python/Pandas?”
I'm also new to Python and discovered NumPy for highly efficient conditional statements. This video was life-changing: [this video]
Recommends DMSBrian · 2023-04 · Stack Overflow · on “Getting date difference between two rows with specific conditions”
You should probably use vector operations for it, it'll run much faster than iloc, map, apply or any sort of loop. Look into numpy.where (or numpy.select if your conditions get long or complex enough). This way you can write your function to essentially operate on the entire column rather than its individual rows (which takes forever) There's a good talk introducing basic examples of using it: [this video] In your case, the first thing that comes to my mind - use groupby in order to get data of particular customers, then simply check if the first date is an activation or suspension and based on that, calculate the total time for each of them by pairing the dates (assuming your system works well,
Recommends chawila · 2024-01 · Stack Overflow · on “How to convert these functions to act on the entire dataframe and speed up my python code”
Thanks for the quick answer, but i was avoiding the explicit `loops` and `maps` to do that because it is taking forever when i loop through 3 month worth of data or more. I tried using `np.apply_along_axis(self.is_uptrend, axis=1, arr=dataframe, df=dataframe)` and it took a lot of time. Which is why i want each function (`is_uptrend()`, `has_bull_choch()`) to use vectorization and return `Dataframe` without using explicite loops and maps. like on this video: [this video]