Pack 456 · Developer stacks · 13 items
Apache Spark Data Engineering Pack
11 prompts + 2 skills for data engineers running Apache Spark and lakehouse pipelines — real jobs from the week, with the steps, rules and output an expert would ask for. 7 written for this work, 6 more from the library.
€4one-time · 13 prompts & skills · ≈ €0.31 each
- Instant download
- Yours to keep
- Works in ChatGPT, Claude, Gemini
What's inside
Made for data engineers running Apache Spark and lakehouse pipelines
6 prompts + 1 skill- PromptApache Spark Job Performance Tuning — Shuffles, Data Skew, Partition Sizing, Joins and Caching
Speeds up slow Spark jobs with evidence from the Spark UI — reading stages and tasks, reducing shuffles, fixing skewed joins, broadcast joins, partition sizing, adaptive query execution settings, caching decisions, file sizes and small-file problems — with before-and-after run times and costs.
- PromptSpark Major Version Upgrade Plan — ANSI SQL Mode, Spark Connect, Deprecated APIs and Testing
Plans an upgrade to a new major Spark version — breaking changes such as stricter ANSI SQL behaviour, removed or deprecated APIs, Spark Connect client mode, library and connector compatibility, data-level regression tests comparing outputs, performance checks and a phased rollout.
- PromptLakehouse Table Design on Spark — Delta Lake, Iceberg or Hudi, Partitioning, Merges and Maintenance
Designs lakehouse tables on Spark — choosing an open table format, partitioning and clustering, merge and upsert patterns, schema evolution, time travel and retention, compaction and clean-up jobs, concurrent writers and costs — for reliable analytics and machine learning data.
- PromptSpark Structured Streaming Pipeline Design — Sources, Watermarks, State, Checkpoints and Reliable Sinks
Builds Structured Streaming pipelines that survive restarts — source configuration, event time and watermarks for late data, stateful aggregations and state size, checkpoint management, idempotent or transactional sinks, trigger choices, monitoring lag and safe changes to running queries.
- PromptPySpark Code Quality — Testable Transformations, Unit and Data Tests, Local Development and CI
Brings software engineering discipline to PySpark code — pure transformation functions, avoiding Python user-defined functions where built-ins exist, schema definitions, unit tests with small local sessions, data quality tests, packaging and CI — so notebooks become maintainable pipelines.
- PromptSpark Cluster and Cost Management — Sizing, Autoscaling, Spot Capacity, Job Clusters and Cost Attribution
Controls Spark compute costs on managed platforms — sizing executors and nodes from workload profiles, job versus interactive clusters, autoscaling limits, spot or preemptible capacity with fallbacks, auto-termination, cost tagging by team and job, and reviewing the most expensive jobs first.
- SkillApache Spark Engineering Partner
A standing Spark partner for data engineers: performance tuning from the Spark UI, major version upgrades, lakehouse tables, streaming pipelines, PySpark code quality and cluster cost control — correct outputs, faster jobs, smaller bills.
Bonus prompts & skills
5 prompts + 1 skill · free with the packFrom the wider PromptCorrectly library, picked for the same work. They come in the same .zip and the start file tells your AI when to use each one.
- SkillExploratory Data Analysis Strategist
Produces a structured EDA plan and narrative findings for any new dataset before modeling.
Library · Free account - PromptData-Pipeline Orchestration Agent Runbook
Operates an agent that runs a multi-stage data pipeline with validation gates, idempotency, and quarantine for bad records.
Library · In a pack - PromptData Quality Monitoring Dashboard Spec
Specifies a data-quality dashboard tracking freshness, completeness, validity, and anomaly checks per pipeline.
Library · In a pack - PromptTranslate Business Questions Into SQL
Turns a plain-English stakeholder question into a correct, well-commented SQL query against a known schema.
Library · Free - PromptRoot-Cause A Metric Spike Or Drop
Drives a systematic decomposition to explain why a metric moved, with SQL to test each hypothesis.
Library · In a pack - PromptReproducible Data Management Plan
Drafts a funder-ready data management plan covering collection, storage, sharing, ethics, and long-term preservation.
Library · In a pack
What you download
Everything comes in one pack (.zip) you keep. Start with 00-GIVE-THIS-TO-YOUR-AI.md: paste it into your AI and it knows which file to use for which job, and what to ask you first.
More for Developers · Data Analysis & Excel
apache-spark-data-engineering/ ├─ 00-GIVE-THIS-TO-YOUR-AI.md ├─ START-HERE.md ├─ TIPS-AND-FOLLOW-UPS.md ├─ AI-CHEAT-SHEET.md ├─ prompts/ │ ├─ 01-apache-spark-job-performance-tuning-shuffles-skew-and-partitions.md │ ├─ 02-spark-4-upgrade-ansi-mode-spark-connect-and-compatibility-plan.md │ ├─ 03-spark-lakehouse-table-design-with-delta-iceberg-or-hudi.md │ ├─ 04-spark-structured-streaming-pipeline-design-and-exactly-once-sinks.md │ ├─ 05-pyspark-code-quality-testing-and-local-development-workflow.md │ ├─ 06-spark-cluster-and-cost-management-on-databricks-emr-or-dataproc.md │ ├─ 07-data-pipeline-orchestration-agent-runbook.md │ ├─ 08-data-quality-monitoring-dashboard-spec.md │ ├─ 09-translate-business-questions-into-sql.md │ ├─ 10-root-cause-a-metric-spike-or-drop.md │ └─ 11-reproducible-data-management-plan.md └─ skills/ ├─ apache-spark-engineering-partner/SKILL.md └─ exploratory-data-analysis-strategist/SKILL.md
A look inside
ROLE: You are a Spark performance engineer. Most slow Spark jobs suffer from a handful of problems — huge shuffles, skewed keys, too many tiny files, spills to disk or pointless caching — and guessing at settings rarely helps, so you read the execution plan and Spark UI …
Questions
What do I get?
A .zip with 11 prompts + 2 skills: 6 prompts + 1 skill written for data engineers running Apache Spark and lakehouse pipelines, plus 5 prompts + 1 skill from the wider library that fit the same work. Each prompt is a ready .md file, each skill a SKILL.md folder, plus 00-GIVE-THIS-TO-YOUR-AI.md (a start file your AI reads first), START-HERE.md, TIPS-AND-FOLLOW-UPS.md (tips and the follow-up questions for every file) and the free AI cheat sheet.
How do I use it?
Open ChatGPT, Claude or Gemini and paste 00-GIVE-THIS-TO-YOUR-AI.md, or attach the folder. It asks a few questions about your work once, then picks the right file for each job and fills in the blanks.
Is it a subscription?
No. €4 once, and the files are yours to keep. Nothing renews.
How do I download it?
Right after checkout, on the confirmation page. Sign in later with the email you paid with and the download button waits for you on this page.
Why this price?
A pack is priced by the items written for its trade: €0.50 a prompt and €1 a skill. The 6 bonus items come free on top. This one is €4 for 13 prompts and skills, about €0.31 an item.