Python Benchmark: Grow pandas DataFrame

Looking for a fast way to "Grow pandas DataFrame" in python? Check below to see see which method is the fastest and uses less memory.

Packages

List packages as it's done in a requirements.txt file. Learn more

1
pandas
 
 
הההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההה
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Setup

Runs before the benchmark, and it is not measured. Good place to import libraries and prepare data. Learn more

1
2
3
4
5
6
import pandas as pd
from random import randint
cities = ["London", "Paris", "Berlin"]
df = pd.DataFrame(columns=["city", "cost", "number"])
 
 
הההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההה
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Code 1

1
2
3
4
5
6
7
8
9
rows = []
for i in range(100):
    rows.append({
        "city": cities[i % 3], 
        "cost": randint(0, 100), 
        "number": i,
    })
df = pd.DataFrame(rows)
 
 
הההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההה
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Code 2

1
2
3
4
5
6
7
for i in range(100):
    row = pd.DataFrame({
        "city": [cities[i % 3]], 
        "cost": [randint(0, 100)], 
        "number": [i],
    })
    df = pd.concat([df, row])
 
 
הההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההה
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Code 3

1
2
3
4
5
6
7
8
9
rows = []
for i in range(100):
    rows.append(pd.DataFrame({
        "city": [cities[i % 3]], 
        "cost": [randint(0, 100)], 
        "number": [i],
    }))
    
df = pd.concat(rows)
 
 
הההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההה
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Code 4

1
2
3
4
5
6
7
for i in range(100):
    row = pd.DataFrame({
        "city": [cities[i % 3]], 
        "cost": [randint(0, 100)], 
        "number": [i],
    })
    df = df.append(row)
 
 
הההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההה
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Teardown

Runs after the benchmark, and it is not measured. Good place to print out results. Learn more

1
print(df)
 
 
הההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההה
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Results

Fastest

Dictionary

4.64 ms

216 op/s

Speed

Benchmark	Runtime	Operations	VS	Visual
Dictionary	4.64 ms	216 op/s	1×	0.00463768164627254
pandas.concat once	150 ms	6.65 op/s	32×	0.15044381795451045
DataFrame.append	331 ms	3.02 op/s	71×	0.33112187194637954
pandas.concat	373 ms	2.68 op/s	80×	0.37318035308271646

Memory

Benchmark	Peak Memory	Score	VS
DataFrame.append	< 1 KB	8.48	1×
pandas.concat once	< 1 KB	8.82	1×
pandas.concat	< 1 KB	8.43	1×
Dictionary	< 1 KB	10.3	1×

Output

Dictionary

      city  cost  number
0   London    36       0
1    Paris    40       1
2   Berlin    29       2
3   London    34       3
4    Paris    16       4
..     ...   ...     ...
95  Berlin    88      95
96  London    34      96
97   Paris    84      97
98  Berlin    76      98
99  London     7      99

[100 rows x 3 columns]

pandas.concat

      city cost number
0   London   90      0
0    Paris   35      1
0   Berlin  100      2
0   London    4      3
0    Paris   68      4
..     ...  ...    ...
0   Berlin   65     95
0   London   45     96
0    Paris   94     97
0   Berlin   24     98
0   London   28     99

[100 rows x 3 columns]

pandas.concat once

      city  cost  number
0   London    26       0
0    Paris    82       1
0   Berlin    49       2
0   London    25       3
0    Paris    19       4
..     ...   ...     ...
0   Berlin     4      95
0   London    37      96
0    Paris    16      97
0   Berlin    65      98
0   London    17      99

[100 rows x 3 columns]

DataFrame.append

      city cost number
0   London   32      0
0    Paris   31      1
0   Berlin   71      2
0   London   10      3
0    Paris   84      4
..     ...  ...    ...
0   Berlin   51     95
0   London   75     96
0    Paris   97     97
0   Berlin   15     98
0   London   50     99

[100 rows x 3 columns]