Python Benchmark: Grow pandas DataFrame

Looking for a fast way to "Grow pandas DataFrame" in python? Check below to see see which method is the fastest and uses less memory.

Packages

List packages as it's done in a requirements.txt file. Learn more

1
pandas
 
 
הההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההה
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Setup

Runs before the benchmark, and it is not measured. Good place to import libraries and prepare data. Learn more

1
2
3
4
5
6
import pandas as pd
from random import randint
cities = ["London", "Paris", "Berlin"]
df = pd.DataFrame(columns=["city", "cost", "number"])
 
 
הההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההה
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Code 1

1
2
3
4
5
6
7
8
9
rows = []
for i in range(100):
    rows.append({
        "city": cities[i % 3], 
        "cost": randint(0, 100), 
        "number": i,
    })
df = pd.DataFrame(rows)
 
 
הההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההה
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Code 2

1
2
3
4
5
6
7
for i in range(100):
    row = pd.DataFrame({
        "city": [cities[i % 3]], 
        "cost": [randint(0, 100)], 
        "number": [i],
    })
    df = pd.concat([df, row])
 
 
הההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההה
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Code 3

1
2
3
4
5
6
7
8
9
rows = []
for i in range(100):
    rows.append(pd.DataFrame({
        "city": [cities[i % 3]], 
        "cost": [randint(0, 100)], 
        "number": [i],
    }))
    
df = pd.concat(rows)
 
 
הההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההה
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Code 4

1
2
3
4
5
6
7
for i in range(100):
    row = pd.DataFrame({
        "city": [cities[i % 3]], 
        "cost": [randint(0, 100)], 
        "number": [i],
    })
    df = df.append(row)
 
 
הההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההה
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Teardown

Runs after the benchmark, and it is not measured. Good place to print out results. Learn more

1
print(df)
 
 
הההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההההה
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Results

Fastest

Dictionary

2.10 ms

476 op/s

Speed

Benchmark	Runtime	Operations	VS	Visual
Dictionary	2.10 ms	476 op/s	1×	0.0021013558443103517
pandas.concat once	159 ms	6.28 op/s	76×	0.1591992722824216
pandas.concat	329 ms	3.04 op/s	157×	0.3288725893944502
DataFrame.append	379 ms	2.64 op/s	180×	0.378707823343575

Memory

Benchmark	Peak Memory	Score	VS	Visual
DataFrame.append	312 KB	5.93	1×	312
Dictionary	476 KB	8.00	1.52×	476
pandas.concat once	696 KB	5.95	2.2×	696
pandas.concat	896 KB	5.53	2.9×	896

Output

Dictionary

      city  cost  number
0   London    32       0
1    Paris    22       1
2   Berlin    64       2
3   London    43       3
4    Paris    79       4
..     ...   ...     ...
95  Berlin    32      95
96  London    57      96
97   Paris    73      97
98  Berlin    88      98
99  London    12      99

[100 rows x 3 columns]

pandas.concat

      city cost number
0   London   62      0
0    Paris   50      1
0   Berlin   63      2
0   London   83      3
0    Paris   88      4
..     ...  ...    ...
0   Berlin   22     95
0   London   96     96
0    Paris    5     97
0   Berlin   30     98
0   London   21     99

[100 rows x 3 columns]

pandas.concat once

      city  cost  number
0   London    87       0
0    Paris    47       1
0   Berlin    71       2
0   London    81       3
0    Paris    43       4
..     ...   ...     ...
0   Berlin    21      95
0   London    15      96
0    Paris    85      97
0   Berlin    76      98
0   London    81      99

[100 rows x 3 columns]

DataFrame.append

      city cost number
0   London   81      0
0    Paris   29      1
0   Berlin   96      2
0   London   97      3
0    Paris   99      4
..     ...  ...    ...
0   Berlin   58     95
0   London    7     96
0    Paris   27     97
0   Berlin   87     98
0   London   17     99

[100 rows x 3 columns]