Deequ

Visit Website

Introduction

Deequ is a library built on top of Apache Spark for defining unit tests for data, which measure data quality in large datasets, according to its README. Python users can use PyDeequ.

Checks declare constraints such as completeness, uniqueness, value ranges and patterns, and Deequ computes the metrics efficiently on Spark. It can suggest constraints by profiling data, store metrics in a repository to track quality over time, and detect anomalies in metric trends.

Deequ 2.1.0 and later require Java 11, and artifacts are built for specific Spark versions. Version 2.1.0 was released in September 2026.

In the quality loop it supports data quality checks in Spark pipelines.

The source code is at awslabs/deequ on GitHub under the Apache-2.0 licence, with about 3,700 stars; the most recent commit was on 16 September 2026.

Features

1. Constraint checks: Completeness, uniqueness, ranges and more.
2. Spark scale: Large datasets.
3. Constraint suggestion: Profile and propose checks.
4. Metrics repository: Track over time.
5. Anomaly detection: Spot metric changes.

Pros

1. Scales with Spark: Built for big data.
2. Free to use: Open source under the Apache-2.0 licence, so the core tool costs nothing to use.
3. Actively maintained: The repository had commits as recent as 16 September 2026.
4. Community: About 3,700 GitHub stars.

Cons

1. Spark required: Heavy for small data.
2. Scala-first: Python via PyDeequ.

Usage and Links (Example)

val result = VerificationSuite().onData(df)
  .addCheck(Check(CheckLevel.Error, "orders").isComplete("id").isUnique("id"))
  .run()

Resources:

• Deequ website

Pricing

Deequ is free and open source under the Apache-2.0 licence.

Recommended for?

1. Spark data teams.
2. Not a fit for small datasets without Spark.

Conclusion

Deequ is a proven data quality library for Spark at scale.

On CheckOps, Deequ is filed under the Integration testing and Production monitoring & observability stages. Related tools listed here include Great Expectations (GX), Soda Core, Pandera, pytest, Hypothesis and Giskard. Before choosing, compare them on licence, supported languages and platforms, hosting model, and how well each fits your existing CI pipeline. Facts on this page were checked against the official website, documentation and repository on 11 November 2026; statements the vendor makes about its own product are marked as vendor claims.