Reading CSVs in Python: pandas, polars, and datatable

I typically use pandas to read CSV files in Python. Anecdotally, most data scientists operating in the Python ecosystem do the same. When using R, I tend to bounce between dplyr or data.table. I always think about giving dtplyr a try to ge the “best of both worlds,” but I haven’t gotten around to it. I just came across the polars Python package today and thought I’d to a quick (and too simple) benchmark comparing pandas, polars (eager and lazy execution), and datatable. This is by no means an exhaustive sample, and I have used others (e.g. csv, dask, siuba) depending on the workflow and size.

import pandas as pd
import polars as pl
import datatable as dt

!wc -l /gpfs52/data/p_matheny_lab/khanms/data/imdb_folds.csv

50001 /gpfs52/data/p_matheny_lab/khanms/data/imdb_folds.csv

The file I use here is 50,000 lines containing mostly text (IMDb movie reviews). I’ll be using the %timeit Jupyter magic command for time benchmarking here.

csv_file = '/gpfs52/data/p_matheny_lab/khanms/data/imdb_folds.csv'

pandas

%%timeit
pandas_df = pd.read_csv(csv_file)

404 ms ± 8.51 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

pd.read_csv(csv_file).head(3)

polars

You can convert a polars dataframe to a pandas dataframe using polars_df.to_pandas().

%%timeit

# Eager execution
polars_df = pl.read_csv(csv_file)

84.2 ms ± 1.88 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

pl.read_csv(csv_file).head(3)

%%timeit

# Lazy execution
polars_df_lazy = pl.scan_csv(csv_file)

765 µs ± 39.6 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

pl.scan_csv(csv_file).head(3)

<polars.internals.lazy_frame.LazyFrame at 0x7f36b2154be0>

# Use `collect` method
pl.scan_csv(csv_file).head(3).collect()

datatable

You can convert a datatable dataframe to a pandas dataframe using datatable_df.to_pandas().

%%timeit
datatable_df = dt.fread(csv_file)

161 ms ± 15.9 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

dt.fread(csv_file).head(3)