Quickstart: DataFusion

Speed up the sorts in your DataFusion queries. kwker_datafusion::register adds one rule to your SessionContext: every ORDER BY on a single number column (with or without LIMIT) then runs on Kwker, and everything else stays DataFusion's own. Your SQL and your results stay the same. About five minutes.

Install

[dependencies]
datafusion = "55"
kwker-datafusion = "0.1"
tokio = { version = "1", features = ["macros", "rt-multi-thread"] }

Keep your SQL

use datafusion::prelude::*;

#[tokio::main]
async fn main() -> datafusion::error::Result<()> {
    let ctx = kwker_datafusion::register(SessionContext::new());
    ctx.register_csv("trips", "trips.csv", CsvReadOptions::new()).await?;
    let top = ctx.sql("SELECT * FROM trips ORDER BY fare DESC LIMIT 100").await?;
    top.show().await?;
    Ok(())
}

EXPLAIN shows KwkerSortExec (or KwkerMergeSortExec, when the table has several partitions) where Kwker took the sort over.

What Kwker takes over

• ORDER BY one column of INT, BIGINT, INT UNSIGNED, BIGINT UNSIGNED, FLOAT or DOUBLE, ascending or descending, NULLS FIRST or NULLS LAST, with or without LIMIT.
• Every other sort (several keys, text, dates) runs on DataFusion's own sort, unchanged.

Notes

• The results equal DataFusion's: the same order, rows with equal keys in their input order.
• Floats follow DataFusion's order: NaN after every number ascending, before them descending; -0.0 before +0.0.
• Kwker sorts on every core by default.

Next steps

• Data and analytics: every data workflow - NumPy, tables, DuckDB, DataFusion, large files.
• Quickstart: DuckDB: the same speed-up inside DuckDB.