Schema-first synthetic data for engineering teams

Synthetic enterprise data from schemas, SQL DDL, dbt metadata, data dictionaries, and generation plans

Name: Great Generator
Author: Ravi Kiran Pagidi

Great Generator helps data teams create realistic non-production datasets from schema contracts, JSON Schema, dbt metadata, data dictionaries, SQL DDL, Spark pipelines, lakehouse demos, CDC, anomalies, relational models, and deterministic advisor-reviewed generation.

Great Generator creates synthetic data. It does not anonymize, mask, de-identify, or transform production records.

Get started View GitHub Read docs Architecture Install from PyPI

Schema-first Pandas and Spark SQL DDL JSON Schema dbt metadata Data dictionaries CDC and anomalies Optional AI advisor Optional MCP server Deterministic generation

Realistic synthetic data for real engineering workflows.

Schema SQL DDL

from great_generator import generate_from_schema

schema = """
customer_id int,
customer_name string,
email string,
signup_date date,
account_status string,
balance decimal(12,2)
"""

df = generate_from_schema(schema=schema, rows=1000, seed=42)
print(df.head())

from great_generator import parse_ddl, generate_from_schema

ddl = """
CREATE TABLE sales.customers (
  customer_id BIGINT PRIMARY KEY,
  customer_name STRING NOT NULL,
  email VARCHAR(120) UNIQUE,
  signup_date DATE,
  balance DECIMAL(12,2)
)
"""

contract = parse_ddl(ddl, dialect="databricks")
df = generate_from_schema(contract, rows=1000, seed=42)

Architecture

One view of the Great Generator platform shape.

This diagram shows how schema ingestion, model understanding, generation, validation, metadata, and delivery fit together for data engineering, QA, analytics, Spark, and lakehouse workflows.

Great Generator enterprise architecture diagram showing users, interfaces, schema ingestion, data generation, validation, observability, output connectors, export formats, and ecosystem integrations. — Great Generator creates synthetic data from schemas and generation specifications, then returns DataFrames or files that teams can validate, observe, and deliver through their own runtime and storage connectors.

Install

Simple install. Optional extras when you need Spark, Delta, dbt/YAML ingestion, advisors, or MCP.

Install with a hyphen. Import with an underscore.

pip install great-generator
pip install "great-generator[spark]"
pip install "great-generator[delta]"
pip install "great-generator[databricks]"
pip install "great-generator[dbt]"
pip install "great-generator[schema-ingest]"
pip install "great-generator[ai]"
pip install "great-generator[anthropic]"
pip install "great-generator[ollama]"
pip install "great-generator[mcp]"
pip install "great-generator[cloud]"
pip install "great-generator[hf]"
pip install "great-generator[transformers]"

import great_generator

from great_generator import generate_from_schema

df = generate_from_schema("id int, name string", rows=100)

What is new

The project has grown beyond simple schema samples.

Recent work adds optional Unity Catalog metadata-driven generation, Contract-to-Test SQL workload compilation, JSON Schema ingestion, dbt metadata ingestion, data dictionary ingestion, query-aware generation, SQL DDL contracts, advisor artifacts, optional MCP tools, Hugging Face Dataset conversion, and stronger launch documentation while keeping the base package lightweight.

Databricks

Unity Catalog metadata generation

Inspect permitted table metadata, discover declared relationships, generate synthetic Pandas or Spark tables, validate joins, save snapshots, and build review-first notebooks without reading source records.

Optional

MCP server

Let MCP-compatible assistants call selected Great Generator tools for local schema generation, DDL parsing, relational data, query coverage, and export.

Current

Query-aware generation

Generate synthetic data that contains required filter values, partition dates, selectivity targets, and relational join paths for SQL and pipeline tests.

Testing

Contract-to-Test compiler

Compile a documented SQL subset into deterministic predicate, join, NULL, and boundary witnesses with explicit coverage denominators and bounded expected outcomes.

Contracts

SQL DDL ingestion

Parse the documented ANSI, Spark, and Databricks `CREATE TABLE` subset into canonical contracts with stable fingerprints and parser diagnostics.

Contracts

JSON Schema ingestion

Generate synthetic DataFrames from API-style JSON Schema contracts using the documented v1 object/property subset.

Analytics

dbt metadata ingestion

Create synthetic model data from common dbt `schema.yml` and `manifest.json` metadata for analytics engineering tests.

Enterprise

Data dictionaries

Load CSV, YAML, or JSON data dictionaries and generate test data from enterprise schema documentation.

Planning

AI advisor layer

Use optional design-time advisors for schema understanding, column tagging, and realism review. Advisors produce JSON artifacts; they do not generate row data.

Optional

Hugging Face integration

Convert generated pandas outputs to Hugging Face Datasets, package related tables as DatasetDict objects, and generate synthetic dataset cards.

Repeatability

Generation plans

Inspect, edit, save, and reuse `GenerationPlan` and `ColumnTags`. The same schema, plan, seed, and arguments are designed to produce repeatable output.

Data systems

Relational and lakehouse data

Generate parent-child tables, CDC records, anomalies, SCD2 history, dimensional models, Data Vault-style examples, and Spark/Delta outputs.

Core workflows

Pick the API that matches your task.

1. Generate from schema

df = generate_from_schema(schema, rows=1000)

Lower-environment data
QA and ETL testing
Analytics prototypes
API contract testing

2. Generate related tables

data = generate_relational(
    tables={
        "customers": {"schema": "customer_id int primary key, customer_name string", "rows": 1000},
        "orders": {"schema": "order_id int primary key, customer_id int references customers.customer_id", "rows": 5000},
    }
)

Joins and referential integrity
Warehouse and lakehouse demos
Dimensional modeling
Integration tests

3. Parse SQL DDL

contract = parse_ddl(ddl, dialect="databricks")
print(contract.fingerprint())

Database contracts
Warehouse schemas
Lakehouse table definitions
Repeatable schema parsing

4. Use an optional advisor

plan = infer_generation_plan(
    "customer_id int, customer_name string, email string",
    advisor="none"
)

df = generate_from_schema(
    "customer_id int, customer_name string, email string",
    rows=100,
    plan=plan,
    seed=42,
)

Design-time review
Column tagging
Realism review
Auditable generation plans

`advisor="none"` is the default and does not call a model.

5. Generate from Unity Catalog metadata

data, report = generate_relational_from_unity_catalog(
    "retail.sales.fact_orders",
    include_related=True,
    output="spark",
    spark=spark,
    return_report=True,
)

Metadata-only source inspection
Declared and explicit relationships
Composite key validation
Review-first notebook generation

Feature set

Practical capabilities for test data, demos, and engineering workflows.

Schema-first generation

Generate synthetic DataFrames from mappings, DDL strings, DataFrames, Spark schemas, `TableSchema`, and domain schemas.

SQL DDL parsing

Use `parse_ddl(...)` for documented SQL `CREATE TABLE` contracts with stable fingerprints.

JSON Schema, dbt, and dictionaries

Generate from API contracts, analytics metadata, and enterprise schema documentation.

Semantic field detection

Recognize names, IDs, contacts, dates, amounts, statuses, quantities, and lifecycle fields from column names.

Query-aware generation

Include required filter values, partition dates, selectivity targets, and join paths when your tests need them.

Contract-to-Test generation

Parse supported SQL workloads and materialize validated Pandas witness rows for predicates, joins, NULL states, and thresholds.

Relational generation

Create parent-child tables with primary keys, foreign keys, and valid references by default.

Unity Catalog metadata

Normalize permitted UC schemas and declared relationships into existing contracts without reading source table records.

CDC simulation

Generate insert, update, and delete style records with event timestamps and ingestion timestamps.

Anomaly injection

Add opt-in nulls, duplicates, orphan keys, late records, outliers, invalid statuses, and negative amounts for data quality tests.

SCD2 history

Create slowly changing dimension history tables for supported Pandas domain workflows.

Dimensional models

Generate facts and dimensions for analytics engineering, BI examples, and warehouse modeling demos.

Data Vault models

Create hubs, links, and satellites for architecture examples and modeling experiments.

Pandas and Spark engines

Use Pandas locally or Spark in runtimes such as Databricks, Fabric Spark, EMR, Glue, and Synapse Spark.

Exports

Use convenience exports for CSV, JSON, Parquet, and Delta, or write returned DataFrames with native APIs.

Recipes and CLI

Package repeatable generation scenarios as JSON, TOML, or simple YAML recipes and run them from the command line.

Advisor artifacts

Create optional `GenerationPlan`, `ColumnTags`, and realism review artifacts before deterministic generation.

Manifest metadata

Record generation parameters, tables, validation checks, schema fingerprints, and optional advisor contribution.

Optional MCP server

Expose selected public APIs to local MCP clients with path checks, row limits, overwrite protection, file outputs, manifests, and previews.

Who this is for

Built for teams that need useful data without production records.

Great Generator is designed for data engineers, QA engineers, analytics engineers, platform teams, Spark users, and developers who need realistic non-production datasets without using production records.

Generate data from schema Generate from JSON Schema Generate from dbt metadata Generate from data dictionaries Generate from SQL DDL contracts Create relational test data Use Spark and Delta workflows Generate from Unity Catalog metadata

Common use cases

Practical synthetic data for engineering, analytics, demos, and lower environments.

Schema-based test data

Generate test data from table schemas, DataFrame contracts, and SQL DDL definitions.

Relational workflows

Create related data for joins, warehouse models, parent-child tests, and query examples.

CDC and anomalies

Build CDC-style events and controlled dirty data for data quality validation.

Spark and lakehouse demos

Create Spark and Delta examples for notebooks, ETL tests, and lakehouse documentation.

Query coverage

Generate data that exercises filters, partitions, selectivity targets, and joins.

Plan review

Review generation plans before producing deterministic synthetic datasets.

Schema inputs

Start from the contract you already have.

Input type	Example	Best for	Status
Plain Python mapping	`{"name": "string", "age": "int"}`	Fast schema-based generation	Supported
Rich mapping and custom rules	`custom_rules={"age": {"min": 18}}`	Business ranges, categories, patterns, and null rates	Supported through `custom_rules`
Pandas dtype mapping	`df.dtypes.to_dict()`	Pandas and notebook workflows	Supported
Pandas DataFrame schema	Empty or populated `DataFrame`	Preserve Pandas column names and dtypes	Supported
Compact DDL string	`"id int, name string"`	SQL-like quick starts	Supported
Full SQL `CREATE TABLE` DDL	`parse_ddl(ddl, dialect="databricks")`	Contracts from databases, warehouses, Spark, and lakehouse tables	Supported for documented subset
PySpark `StructType`	`StructType([...])`	Spark notebook schemas	Supported
PySpark DataFrame	Empty or populated Spark DataFrame	Use an existing Spark schema	Supported
`TableSchema`	`TableSchema(name="customers", ...)`	Library-native schema metadata	Supported
`DomainSchema`	Multi-table schema metadata	Relational and domain-shaped datasets	Supported
JSON Schema	`schema.json`	Application contracts	Supported
dbt schema.yml	`models/schema.yml`	Analytics engineering contracts	Supported
dbt manifest.json	`target/manifest.json`	Compiled dbt metadata	Supported
Data dictionary	`data_dictionary.csv`	Enterprise schema docs	Supported
Pydantic model	`BaseModel`	API model contracts	Planned
Dataclass	`@dataclass`	Python application models	Planned

Query-aware generation

Generate data that your SQL query can actually match.

Query-aware generation creates synthetic data that contains the required values, partition dates, and join paths your SQL queries expect.

All query-aware options are optional. Existing behavior is unchanged unless users provide `required_values`, `partition_by`, `target_selectivity`, `ensure_join_coverage`, or `query_profile`.

Query-aware generation helps synthetic data match expected query values, partition dates, and join paths. It does not guarantee identical production performance because file layout, table statistics, clustering, caching, concurrency, warehouse size, and query engine configuration also affect runtime.

Current query-aware shaping is implemented for Pandas generation paths. Spark-native query-aware generation is planned; Spark users can continue using existing Spark generation paths and normal Spark writers.

df = generate_from_schema(
    schema=schema,
    rows=100000,
    required_values={
        "region": ["SOUTH"],
        "product_type": ["CHECKING", "SAVINGS"],
    },
    partition_by={
        "column": "business_date",
        "values": ["2026-01-01", "2026-01-02"],
        "distribution": "balanced",
    },
    target_selectivity={"region": {"SOUTH": 0.25}},
)

SQL DDL and contracts

Contract-first schema generation from documented SQL DDL.

Use `parse_ddl(...)` to parse documented SQL `CREATE TABLE` DDL into canonical contracts with stable fingerprints, table names, column order, normalized types, keys, relationships, constraints, comments, defaults, and selected Spark/Databricks metadata where supported.

The parser supports a documented subset of ANSI, Spark, and Databricks DDL. Unsupported syntax should produce clear diagnostics rather than silent assumptions.

from great_generator import parse_ddl, generate_from_schema

contract = parse_ddl(ddl, dialect="databricks", strict=True)
print(contract.fingerprint())

df = generate_from_schema(contract, rows=1000, seed=42)

Optional AI advisor layer

Design-time advice, deterministic row generation.

The advisor layer can propose a generation plan, tag columns, and review a sample for realism. Advisors do not generate row data. Generation remains deterministic because the generator consumes inspectable JSON artifacts, not model output at row-generation time.

schemaadvisorplanreviewgenerationdata

Advisor	Status	Network	Notes
none	default	none	No API key, no model call.
Anthropic	optional	online	Requires advisor extra and API key.
Ollama	optional	local	Supports offline local model use.
OpenAI	stub/planned	online	Reserved interface; not listed as active advisor support.
llama.cpp	stub/planned	local	Reserved interface; not listed as active advisor support.

Relational and lakehouse workflows

Generate connected tables, history, CDC, and modeling examples.

Great Generator supports parent-child tables, primary keys and foreign keys, fact and dimension examples, CDC simulation, anomaly injection, SCD2 history, dimensional models, Data Vault-style examples, and Spark/Delta output.

Returned DataFrames stay in your control, so you can write to files, catalogs, databases, or cloud storage through native Pandas and Spark APIs.

data = generate_relational(tables=tables, seed=42)

customers = data["customers"]
orders = data["orders"]

data = generate_domain("banking", history="scd2")
cdc = generate_cdc("banking", table="customers", rows=1000)

Spark, Delta, and cloud paths

Use the same package locally and in Spark notebooks.

Generate Pandas or Spark DataFrames and write through normal runtime APIs. Pandas can write local CSV, JSON, and Parquet. Spark can write to DBFS, ADLS, S3, GCS, HDFS, mounted paths, Parquet, Delta, and configured database connectors.

df.to_parquet("customers.parquet", index=False)

spark_df.write.mode("overwrite").parquet("s3://bucket/demo/customers")
spark_df.write.format("delta").mode("overwrite").save("dbfs:/tmp/demo_delta")

Platform examples

Notebook-ready examples for local, lakehouse, warehouse, and cloud storage workflows.

Great Generator stays DataFrame-first: generate synthetic data, keep the DataFrame, and write it with your runtime's native APIs or the optional helper when it fits.

Local

Pandas and Jupyter

Run schema, JSON Schema, dbt, data dictionary, relational, query-aware, CDC, and anomaly notebooks locally.

Open examples

Lakehouse

Databricks and Delta

Generate Spark and Delta data, use Unity Catalog metadata as an optional schema source, validate related tables, and create review-first sandbox notebooks.

Open examples

Lakehouse

Microsoft Fabric

Create synthetic data in Fabric Spark notebooks and write Lakehouse tables or OneLake Files outputs.

Open examples

Warehouse

Snowflake

Generate Parquet files and load with user-managed stage and COPY INTO SQL templates.

Open examples

Storage

S3, ADLS, and GCS

Use native Pandas/Spark writers or optional cloud helper paths with credentials managed by your environment.

Open examples

Great Generator does not create cloud resources, configure IAM, manage secrets, or claim universal benchmark performance.

Synthetic benchmark harness

Generate repeatable datasets and SQL templates for your own environment.

The benchmark harness starts with an implemented retail star schema and query templates for partition filters, joins, selectivity filters, rollups, and large group-by behavior.

These examples are not universal performance claims. Results depend on warehouse size, cluster configuration, storage layout, file format, statistics, clustering, caching, concurrency, and query engine settings.

python benchmarks/datasets/retail_star_schema/generate_dataset.py \
  --output output/retail_star_schema \
  --format parquet \
  --overwrite

benchmarks/queries/retail_star_schema/q01_partition_filter.sql
benchmarks/results_template/benchmark_results_template.csv

Optional MCP server

Let assistants generate local synthetic datasets without returning large data in chat.

Install `great-generator[mcp]` to expose selected Great Generator tools over local stdio MCP. The server wraps existing public APIs, writes generated datasets to local files, and returns summaries, previews, file paths, warnings, and manifest information.

The MCP server does not introduce a new generation engine, does not call external APIs by default, does not send data to production systems, and does not overwrite files unless `overwrite=True`.

pip install "great-generator[mcp]"

great-generator-mcp

{
  "mcpServers": {
    "great-generator": {
      "command": "great-generator-mcp",
      "args": []
    }
  }
}

Trust, safety, and determinism

Repeatable generated data without production-record transformation.

Great Generator separates design-time advice from row generation. Advisor outputs are saved as JSON artifacts. The generator consumes those artifacts deterministically. This keeps test data repeatable and makes plan review possible before data is generated.

Deterministic path

Same schema, plan, seed, and arguments should produce the same output for repeatable tests and demos.

Offline default

`advisor="none"` is the default. It does not call a model, read API keys, or require network access.

JSON artifacts

Plans, tags, reports, and manifests can be reviewed, edited, saved, and committed with your project.

Synthetic boundary

The library creates synthetic data. It does not anonymize, mask, de-identify, or transform production records.

Documentation

Guides for quick starts, implementation, demos, and releases.

QuickstartInstall, import, generate, inspect, and export. InstallationCore package, Spark, Delta, advisor, and MCP extras. Schema InputsCurrent support matrix, examples, and limitations. JSON SchemaGenerate from API-style JSON Schema contracts. dbt IntegrationGenerate from dbt schema.yml and manifest.json metadata. Data DictionaryGenerate from CSV, YAML, or JSON schema documentation. SQL DDLCanonical contracts, parser diagnostics, and DDL examples. MCP ServerOptional local MCP tools, client configs, safety limits, outputs, and manifests. Query-Aware GenerationRequired values, partitions, selectivity, join coverage, and coverage reports. Contract-to-TestSQL workload parsing, deterministic coverage witnesses, validation, reports, and current limitations. API ReferenceMain public APIs and usage patterns. AI AdvisorsOptional schema planning, column tagging, caching, and offline Ollama usage. Hugging Face IntegrationDataset conversion, DatasetDict helpers, dataset cards, and responsible publishing guidance. Transformers AdvisorOptional local Transformers model planning with deterministic row generation boundaries. Generation Plans`GenerationPlan`, `ColumnTags`, and realism report artifacts. CDCChange data capture style event records. AnomaliesOpt-in data problems for quality testing. Spark and DeltaCluster usage and lakehouse outputs. Unity CatalogMetadata-only inspection, relational generation, snapshots, CDC, validation, and safe notebook generation. Platform ExamplesPandas, Databricks, Fabric, Snowflake, and cloud storage workflows. Cloud ExportsLocal, S3, ADLS Gen2, GCS, optional helper, and credential guidance. Benchmark HarnessRepeatable retail star-schema data, SQL query templates, and result templates. ExamplesNotebook, Spark, assistant, and demo workflow starters. RoadmapPlanned schema adapters, Spark-native work, and contribution areas. Cloud DeploymentAWS, Azure, GCP, Databricks, and storage guidance. Recipes and CLIReusable generation parameters and command-line workflows. Domain PacksSupported industry domains and table layouts. Demo GuideSpeaker-friendly story, examples, and demo flow.

Release highlights

Recent project direction.

Version	Focus	Highlights
0.1.8	Schema ingestion, optional MCP, SEO, org links, and architecture	JSON Schema, dbt metadata, CSV/YAML/JSON data dictionaries, query-aware examples, `great-generator[mcp]`, local-file tools, safety controls, tests, sitemap, robots.txt, SEO metadata, GreatDataLabs links, and architecture diagram.
0.1.7	SQL DDL ingestion and query-aware generation	`parse_ddl(...)`, canonical contracts, parser diagnostics, documented ANSI/Spark/Databricks subset, required values, partitions, selectivity, and join coverage.
0.1.6	AI advisor planning layer	Advisors, `GenerationPlan`, `ColumnTags`, cached advisor calls, manifest metadata, and `plan=` support.
0.1.5	Schema-first docs and Spark writes	Schema input matrix, Databricks examples, Snowflake writes, Azure SQL writes, and documentation site updates.
0.1.1	Advanced APIs	Anomaly labels, SCD2 history, recipes, CLI, dimensional models, and Data Vault models.
0.1.0	Initial release	Domain packs, Pandas/Spark engines, exports, CDC, anomalies, schema generation, and relational generation.

Roadmap

Planned areas are labeled clearly.

Infer query profiles from SQL text OpenAPI and Avro schema inputs Pydantic and dataclass schema adapters Spark-native query-aware generation More Spark and lakehouse examples Quality-tool examples for Great Expectations and Pandera More domain packs and reference values

Project

Created and maintained by Ravi Kiran Pagidi.

Great Generator is MIT licensed and maintained by Ravi Kiran Pagidi.

GitHub repository PyPI package GitHub Wiki Issues License