Building a Self-Healing Data Pipeline on AWS: How We Used Amazon Bedrock to Diagnose and Auto-Remediate ETL Failures

The 2am Problem: Why Data Pipeline Failures Are the Most Expensive Operational Problem Nobody Has Solved

Every data engineering team has a version of the same story. A Glue job fails at 2am. CloudWatch fires an alarm. A data engineer gets paged. They log in, read the error logs, identify the root cause a schema change upstream, a malformed record in the source file, a partition that doesn't exist yet — apply the fix, rerun the job, and go back to sleep. The whole incident takes 45 minutes and costs the organization one disrupted night and one day of delayed downstream reports.

Then it happens again the following Tuesday. Different cause, same pattern: alert, investigate, fix, rerun. And the Tuesday after that.

The problem is not that data pipelines break — they do, and they always will. Source systems change schemas without warning. Files arrive with encoding anomalies. APIs return unexpected null values. Partitions lag behind expectations. These are not exceptional events; they are the normal operating conditions of an enterprise data platform. The problem is that the response to every failure is manual and repetitive — the same categories of errors diagnosed by hand, the same fixes applied, the same engineer paged for problems that could have been resolved autonomously.

When we engaged with an enterprise FMCG company running seventeen AWS Glue ETL jobs feeding a SageMaker-powered demand forecasting platform, the data engineering team was spending approximately 22% of their sprint capacity on pipeline failure investigation and remediation. Not on improving the pipelines. Not on building new data products. On keeping the existing ones running.

The failures fell into five categories that accounted for 84% of all incidents: schema drift from source systems, malformed records in file ingestion, S3 partition path mismatches, Glue job timeout from unexpectedly large datasets, and downstream dependency failures where a job ran successfully but its output was missing a table partition the next job expected.

This post documents the self-healing pipeline architecture we built to handle those five categories autonomously — using Amazon EventBridge to capture Glue job failures, AWS Lambda to collect diagnostic context, Amazon Bedrock to classify the root cause and generate a remediation plan, and Step Functions to execute the fix and retry — with escalation to the data engineering team only for failure classes the system cannot resolve.

As Technical Architect for this initiative at AeonX Digital, I designed the remediation framework, defined the failure taxonomy, and led implementation with the data engineering team. What follows covers the architecture decisions, the implementation details, and the failure modes we did not anticipate until production.

The outcome: autonomous remediation rate of 71% across all pipeline failures, mean time to recovery reduced from 47 minutes to 8 minutes for auto-remediated incidents, and data engineering team capacity redirected from incident response to pipeline development.

Why This Matters Now: The Data Reliability Inflection Point

Three forces are making pipeline self-healing a strategic priority rather than a nice-to-have:

• AI-dependent business processes have zero tolerance for stale data — When a demand forecasting model runs on yesterday's data because the ETL failed overnight, the business impact is direct and measurable: procurement decisions made on outdated signals, inventory positions that are already wrong before the day begins. As AI systems take on more operational decisions, the data pipelines feeding them become critical infrastructure — and critical infrastructure requires operational maturity that manual incident response cannot provide.
• Source system change velocity is accelerating — ERP upgrades, CRM migrations, API version changes, and upstream application deployments routinely introduce schema changes that break downstream pipelines. Five years ago, a major source system change happened annually. Today, with agile development and continuous deployment across the enterprise technology stack, schema-breaking changes happen monthly or weekly. Data pipelines that require manual intervention on every schema change cannot keep pace.
• Generative AI has made root cause diagnosis tractable at scale — The barrier to pipeline self-healing was never the remediation logic — adding a column, adjusting a partition path, increasing a job timeout are all straightforward operations. The barrier was root cause classification: determining from a stack trace and job logs which of the twenty possible failure categories this specific error belongs to, and therefore which remediation to apply. Foundation models are now accurate enough on structured log analysis to automate that classification reliably.

The decision to build this system was driven by a data platform lead who had tracked the team's incident response time over six months, presented the 22% sprint capacity figure to engineering leadership, and received approval to invest in automation that would give that capacity back.

The Business Problem

The data engineering team's pipeline operations had:

• Seventeen Glue ETL jobs running on daily and hourly schedules feeding a demand forecasting platform, a sales analytics dashboard, and a procurement intelligence system
• No automated root cause classification — every failure required a human to read logs and diagnose the cause
• No remediation automation — fixes applied manually by the engineer who investigated the failure
• Inconsistent documentation — the same failure category fixed by different engineers in different ways, with no shared runbook
• No pattern detection — recurring failures from the same root cause (e.g., a source system that changed its date format every time it was upgraded) were treated as independent incidents each time
• Downstream impact visibility limited — when a Glue job failed, the impact on dependent jobs and downstream consumers was not automatically assessed

Business impact:

• 22% of data engineering sprint capacity consumed by pipeline incident response — approximately 1.8 engineer-days per sprint
• Average time to recovery of 47 minutes per incident, including alert-to-response lag at off-hours
• Three incidents in six months where a pipeline failure was not detected until a business stakeholder noticed missing data in a dashboard — the monitoring had not caught the failure early enough
• Demand forecasting model retrained on incomplete data twice due to undetected upstream pipeline failures, producing degraded predictions that persisted for two weeks before detection

The goal was not to eliminate all manual intervention — some failure classes genuinely require human judgment. The goal was to eliminate manual intervention for the failure classes that do not, and to dramatically improve the quality of escalations for the ones that do.

Technical Architecture

Figure 1: Self-Healing Data Pipeline — Closed-Loop Failure Detection, Bedrock Diagnosis, and Auto-Remediation on AWS

AWS Services Used:

• AWS Glue — ETL job execution, Data Catalog for schema management
• Amazon EventBridge — Glue job state change events (FAILED, TIMEOUT) routing to remediation workflow
• AWS Lambda — log collection, context assembly, remediation execution, dependency impact assessment
• Amazon Bedrock (Claude 3 Sonnet) — root cause classification and remediation plan generation
• AWS Step Functions — remediation workflow orchestration with branching, retry logic, and escalation
• Amazon S3 — pipeline artifacts, job scripts, remediation audit log
• AWS Glue Data Catalog — schema version history for drift detection
• Amazon CloudWatch — enhanced job monitoring with custom metrics and anomaly detection
• Amazon SNS — escalation notifications to data engineering team with pre-built investigation context
• Amazon DynamoDB — failure pattern registry, remediation history, and recurrence tracking
• AWS Secrets Manager — source system credentials for schema inspection

The architecture operates as a closed-loop system: Glue job failure triggers EventBridge, which invokes the diagnostic Lambda, which calls Bedrock for classification, which triggers Step Functions for the appropriate remediation branch, which retries the job and records the outcome. The data engineering team is involved only when the system cannot resolve the failure autonomously — and when it does escalate, it sends a structured briefing rather than a raw error notification.

Key Architectural Decisions

These are the decisions that shaped the self-healing system — and the reasoning behind each one.

Decision 1: Why Bedrock for Root Cause Classification Instead of a Rule-Based Classifier?

The first design proposal was a rule-based classifier: a set of regex patterns matched against Glue error messages to categorize failures into the five known classes. It was simple to implement and would have handled the majority of cases correctly.

The problem with a rule-based classifier is brittleness. Glue error messages are not standardized. The same root cause — a schema drift — produces different error text depending on whether the job uses the Glue Data Catalog, reads directly from S3, processes a JDBC source, or fails during a DynamicFrame resolution. Writing regex rules that reliably cover all variants of each failure class is a maintenance burden that grows with every new data source onboarded.

Bedrock classifies failure root causes differently: it reads the full error message, the relevant section of job logs, the job script context, and the Data Catalog schema history, and reasons across all of them to produce a classification with a confidence score and an explanation. It handles novel error message formats without rule updates. It catches failure patterns that combine elements of multiple categories — a schema drift that also caused a record count anomaly — that a single-category rule engine would misclassify.

Approach	Handles Known Failure Patterns	Handles Novel Variants	Maintenance Burden	Explanation Quality
Regex rule engine	Well	Poorly — misclassifies variants	High — rules for every source	None — category label only
ML classifier (trained)	Well	Depends on training data	Medium — retraining on new patterns	None — category label only
Bedrock (LLM reasoning)	Well	Well — generalizes across variants	Low — prompt updates only	High — natural language rationale

The business decision: The maintenance burden of a rule-based classifier grows linearly with data source count. At seventeen Glue jobs across eleven source systems, the rule set was already unwieldy. Bedrock's classification quality was sufficient for production use after prompt calibration on three months of historical failures, and its maintenance burden is a prompt update rather than a rule rewrite.

Decision 2: Why Step Functions for Remediation Orchestration Instead of a Single Lambda?

Each remediation class requires a different sequence of operations. Schema drift remediation requires reading the new schema from the source, updating the Glue Data Catalog, optionally updating the job script if the column is referenced explicitly, and retrying the job. Partition path mismatch remediation requires checking which partitions actually exist in S3, updating the partition metadata in the Data Catalog, and retrying. Timeout remediation requires adjusting the DPU allocation or splitting the job, updating the job definition, and retrying.

A single Lambda function handling all five remediation paths would be a monolithic function with complex branching logic, no visibility into which step failed, and no clean retry behavior at the step level. Step Functions gives each remediation class its own state machine path with step-level error handling, retry configuration, and a clear execution history that makes post-incident review straightforward.

The business decision: Step Functions adds minimal cost at this event volume — approximately $0.25/month for the workflow executions generated by seventeen jobs failing with a realistic frequency. The operational visibility and step-level retry logic it provides are worth far more than that.

Decision 3: Why Auto-Remediate Some Failure Classes and Escalate Others?

Not all failure classes are safe to remediate autonomously. Schema drift remediation — adding a new column to the Data Catalog — is safe to automate: the worst outcome is that a downstream query returns an unexpected column it ignores. Dropping a column from the Data Catalog based on a misclassification is not safe to automate: the worst outcome is that downstream consumers lose data they depend on.

We defined three tiers of automated response based on risk:

Tier	Failure Classes	Automated Action	Human Involvement
Auto-remediate	Partition path mismatch, job timeout (DPU increase), missing output partition	Fix + retry automatically	None unless retry fails
Auto-remediate with notification	Schema additive drift (new column), malformed record skip	Fix + retry + notify data owner	Informed, not required to act
Escalate with context	Schema destructive drift (dropped/renamed column), unclassified failures, third consecutive failure of same class	Collect full context + notify	Required to resolve

The business decision: Autonomous remediation that occasionally makes a wrong decision in the safe tier is acceptable. Autonomous remediation that makes a wrong decision in the destructive tier is not. The tier classification is the risk management layer that makes the system trustworthy enough to run in production without constant oversight.

Implementation Pattern

EventBridge Rule: Capturing Glue Job Failures

# EventBridge rule — captures Glue job state changes to FAILED or TIMEOUT
# Deployed via AWS CDK or CloudFormation

GLUE_FAILURE_RULE = {
    "source": ["aws.glue"],
    "detail-type": ["Glue Job State Change"],
    "detail": {
        "state": ["FAILED", "TIMEOUT"],
        # Scope to production jobs only — exclude dev/test prefixes
        "jobName": [{"prefix": "prod-"}]
    }
}

# EventBridge target: Lambda diagnostic function
# The rule passes the full event detail including jobName, jobRunId,
# error message, and execution time to the diagnostic Lambda

Diagnostic Lambda: Log Collection and Bedrock Classification

import boto3
import json
import time
from datetime import datetime, timezone, timedelta

glue       = boto3.client("glue",            region_name="ap-south-1")
logs       = boto3.client("logs",            region_name="ap-south-1")
bedrock    = boto3.client("bedrock-runtime", region_name="ap-south-1")
sfn        = boto3.client("stepfunctions",   region_name="ap-south-1")
dynamodb   = boto3.resource("dynamodb",      region_name="ap-south-1")

history_table = dynamodb.Table("PipelineFailureHistory")
SFN_ARN       = "arn:aws:states:ap-south-1:111122223333:stateMachine:PipelineRemediationSM"

FAILURE_CLASSES = [
    "SCHEMA_DRIFT_ADDITIVE",      # New column added in source
    "SCHEMA_DRIFT_DESTRUCTIVE",   # Column dropped or renamed in source
    "MALFORMED_RECORDS",          # Unparseable records in source file
    "PARTITION_PATH_MISMATCH",    # S3 partition key/value mismatch
    "JOB_TIMEOUT",                # Job exceeded max runtime
    "DEPENDENCY_FAILURE",         # Upstream job output missing
    "RESOURCE_CONSTRAINT",        # Insufficient DPUs or memory
    "UNCLASSIFIED"                # Cannot confidently determine root cause
]

def diagnose_failure(event, context):
    """
    Called by EventBridge on every Glue job FAILED or TIMEOUT event.
    Collects diagnostic context, classifies root cause via Bedrock,
    and triggers the appropriate Step Functions remediation branch.
    """
    detail   = event.get("detail", {})
    job_name = detail.get("jobName")
    run_id   = detail.get("jobRunId")
    state    = detail.get("state")

    # ── 1. Fetch job run details and error message ────────────────────────
    run_details = glue.get_job_run(JobName=job_name, RunId=run_id)["JobRun"]
    error_msg   = run_details.get("ErrorMessage", "No error message available")
    exec_time_s = run_details.get("ExecutionTime", 0)
    dpu_used    = run_details.get("AllocatedCapacity", 0)

    # ── 2. Collect relevant CloudWatch log lines (last 200 lines) ────────
    log_lines = _get_job_logs(job_name, run_id, max_lines=200)

    # ── 3. Get schema history from Glue Data Catalog ──────────────────────
    schema_context = _get_schema_context(job_name)

    # ── 4. Check recurrence — is this the same failure seen before? ───────
    recent_failures = _get_recent_failures(job_name, hours=72)
    recurrence_note = (
        f"This job has failed {len(recent_failures)} times in the last 72 hours. "
        f"Previous failure classes: {[f['failure_class'] for f in recent_failures]}."
        if recent_failures else "No recent failures for this job."
    )

    # ── 5. Classify root cause via Bedrock ────────────────────────────────
    classification = _classify_with_bedrock(
        job_name=job_name,
        error_msg=error_msg,
        log_excerpt="\n".join(log_lines[-50:]),   # Last 50 lines most relevant
        schema_context=schema_context,
        exec_time_s=exec_time_s,
        dpu_used=dpu_used,
        recurrence_note=recurrence_note
    )

    # ── 6. Record failure in history ──────────────────────────────────────
    history_table.put_item(Item={
        "job_name":       job_name,
        "run_id":         run_id,
        "failure_class":  classification["failure_class"],
        "confidence":     str(classification["confidence"]),
        "bedrock_reason": classification["reasoning"],
        "timestamp":      datetime.now(timezone.utc).isoformat(),
        "state":          state,
        "ttl":            int(time.time()) + (90 * 86400)
    })

    # ── 7. Trigger Step Functions remediation workflow ────────────────────
    sfn.start_execution(
        stateMachineArn=SFN_ARN,
        name=f"remediate-{job_name}-{run_id[:8]}-{int(time.time())}",
        input=json.dumps({
            "job_name":       job_name,
            "run_id":         run_id,
            "failure_class":  classification["failure_class"],
            "confidence":     classification["confidence"],
            "reasoning":      classification["reasoning"],
            "remediation":    classification["remediation_plan"],
            "error_message":  error_msg,
            "recurrence_count": len(recent_failures),
            "schema_context": schema_context
        })
    )

    return {"statusCode": 200, "body": json.dumps(classification)}


def _classify_with_bedrock(job_name, error_msg, log_excerpt,
                            schema_context, exec_time_s, dpu_used,
                            recurrence_note) -> dict:
    prompt = f"""You are a data engineering expert specializing in AWS Glue ETL failures.

Classify the following Glue job failure into exactly one of these categories:
{json.dumps(FAILURE_CLASSES, indent=2)}

JOB CONTEXT:
- Job name: {job_name}
- Execution time: {exec_time_s}s
- DPUs allocated: {dpu_used}
- Recurrence: {recurrence_note}

ERROR MESSAGE:
{error_msg}

RELEVANT LOG LINES (last 50):
{log_excerpt}

SCHEMA CONTEXT (Data Catalog recent changes):
{json.dumps(schema_context, indent=2)}

Respond as a JSON object with these exact keys:
- failure_class: one of the categories above (string)
- confidence: your confidence in this classification, 0.0 to 1.0 (number)
- reasoning: 2-3 sentence explanation of why this is the root cause (string)
- remediation_plan: specific steps to fix this failure (string)
- safe_to_automate: whether this failure class is safe to auto-remediate (boolean)
- estimated_fix_time_minutes: how long the automated fix should take (number)"""

    response = bedrock.invoke_model(
        modelId="anthropic.claude-3-sonnet-20240229-v1:0",
        contentType="application/json",
        accept="application/json",
        body=json.dumps({
            "anthropic_version": "bedrock-2023-05-31",
            "max_tokens": 512,
            "temperature": 0.1,    # Very low — classification needs consistency
            "messages": [{"role": "user", "content": prompt}]
        })
    )
    result = json.loads(response["body"].read())
    return json.loads(result["content"][0]["text"])


def _get_job_logs(job_name: str, run_id: str, max_lines: int = 200) -> list:
    """Fetch Glue job logs from CloudWatch Logs."""
    log_group = f"/aws-glue/jobs/error"
    try:
        response = logs.filter_log_events(
            logGroupName=log_group,
            logStreamNames=[f"{job_name}_{run_id}"],
            limit=max_lines
        )
        return [e["message"] for e in response.get("events", [])]
    except logs.exceptions.ResourceNotFoundException:
        return [f"Log stream not found for run {run_id}"]


def _get_schema_context(job_name: str) -> dict:
    """Get recent schema changes from Glue Data Catalog for this job's tables."""
    try:
        # Fetch job definition to find source/target tables
        job_def   = glue.get_job(JobName=job_name)["Job"]
        args      = job_def.get("DefaultArguments", {})
        db_name   = args.get("--source_database", "default")
        table_name = args.get("--source_table", "")

        if not table_name:
            return {"note": "Could not determine source table from job arguments"}

        table = glue.get_table(DatabaseName=db_name, Name=table_name)["Table"]
        return {
            "table":            f"{db_name}.{table_name}",
            "column_count":     len(table.get("StorageDescriptor", {})
                                    .get("Columns", [])),
            "last_updated":     str(table.get("UpdateTime", "")),
            "partition_keys":   [k["Name"] for k in table.get("PartitionKeys", [])]
        }
    except Exception as e:
        return {"error": str(e)}


def _get_recent_failures(job_name: str, hours: int = 72) -> list:
    """Check failure history for recurrence pattern."""
    cutoff = (datetime.now(timezone.utc) - timedelta(hours=hours)).isoformat()
    response = history_table.query(
        KeyConditionExpression="job_name = :j AND #ts > :t",
        ExpressionAttributeNames={"#ts": "timestamp"},
        ExpressionAttributeValues={":j": job_name, ":t": cutoff}
    )
    return response.get("Items", [])

Step Functions: Remediation State Machine

The state machine receives the Bedrock classification and routes to the appropriate remediation branch. Each branch handles one failure class with its own sequence of fix steps, retry logic, and success/failure outcomes.

START → CheckFailureClass
    ├── PARTITION_PATH_MISMATCH → RepairPartitions → RetryJob → RecordOutcome
    ├── JOB_TIMEOUT             → IncreaseDPU      → RetryJob → RecordOutcome
    ├── MALFORMED_RECORDS       → EnableSkipErrors  → RetryJob → RecordOutcome
    ├── SCHEMA_DRIFT_ADDITIVE   → UpdateDataCatalog → NotifyDataOwner
    │                                              → RetryJob → RecordOutcome
    ├── DEPENDENCY_FAILURE      → CheckUpstreamJob → WaitForUpstream (15 min)
    │                                              → RetryJob → RecordOutcome
    ├── SCHEMA_DRIFT_DESTRUCTIVE → AssembleEscalationContext → EscalateToTeam
    ├── UNCLASSIFIED             → AssembleEscalationContext → EscalateToTeam
    └── ThirdConsecutiveFailure  → AssembleEscalationContext → EscalateToTeam

RetryJob:
    → SUCCEEDED → RecordAutoRemediation (DynamoDB) → END
    → FAILED    → AssembleEscalationContext → EscalateToTeam → END

Partition Repair Lambda: Auto-Remediation Example

import boto3
import json

glue = boto3.client("glue",   region_name="ap-south-1")
s3   = boto3.client("s3",     region_name="ap-south-1")

def repair_partitions(event, context):
    """
    Remediation tool for PARTITION_PATH_MISMATCH failures.
    Scans actual S3 partition paths and syncs them to the Glue Data Catalog.
    """
    job_name      = event["job_name"]
    schema_context = event.get("schema_context", {})

    # Parse database and table from schema context
    table_ref = schema_context.get("table", "")
    if "." not in table_ref:
        return {"status": "SKIPPED", "reason": "Could not determine table from context"}

    db_name, table_name = table_ref.split(".", 1)

    # Get table definition to find S3 location and partition keys
    table     = glue.get_table(DatabaseName=db_name, Name=table_name)["Table"]
    s3_loc    = table["StorageDescriptor"]["Location"]   # e.g. s3://bucket/prefix/
    part_keys = [k["Name"] for k in table.get("PartitionKeys", [])]

    if not part_keys:
        return {"status": "SKIPPED", "reason": "Table has no partition keys"}

    # Parse bucket and prefix from S3 location
    s3_loc_clean = s3_loc.replace("s3://", "")
    bucket, prefix = s3_loc_clean.split("/", 1)

    # List actual partitions present in S3
    paginator = s3.get_paginator("list_objects_v2")
    s3_prefixes = set()

    for page in paginator.paginate(Bucket=bucket, Prefix=prefix, Delimiter="/"):
        for p in page.get("CommonPrefixes", []):
            s3_prefixes.add(p["Prefix"].replace(prefix, "").strip("/"))

    # Get partitions currently registered in Glue
    existing_parts = set()
    paginator_glue = glue.get_paginator("get_partitions")
    for page in paginator_glue.paginate(DatabaseName=db_name, TableName=table_name):
        for part in page.get("Partitions", []):
            existing_parts.add("/".join(part["Values"]))

    # Register partitions that exist in S3 but not in Glue
    new_partitions = []
    for prefix_path in s3_prefixes:
        part_str = prefix_path.replace("=", "/").strip("/")
        if part_str not in existing_parts:
            parts = prefix_path.split("/")
            values = [p.split("=")[-1] for p in parts if "=" in p]
            if len(values) == len(part_keys):
                new_partitions.append({
                    "Values": values,
                    "StorageDescriptor": {
                        **table["StorageDescriptor"],
                        "Location": f"s3://{bucket}/{prefix}{prefix_path}"
                    }
                })

    if new_partitions:
        # batch_create_partition accepts max 25 at a time
        for i in range(0, len(new_partitions), 25):
            batch = new_partitions[i:i+25]
            glue.batch_create_partition(
                DatabaseName=db_name,
                TableName=table_name,
                PartitionInputList=batch
            )

    return {
        "status":           "REPAIRED",
        "partitions_added": len(new_partitions),
        "table":            table_ref,
        "message":          (f"Registered {len(new_partitions)} missing partitions "
                             f"from S3 into Glue Data Catalog for {table_ref}.")
    }

Cost Architecture and AWS Infrastructure Spend

The self-healing system activates only on failure events — it consumes no resources when pipelines are running normally. At the observed failure rate of approximately 38 incidents per month across seventeen jobs:

Service	Usage	Estimated Monthly Cost
Amazon Bedrock (Claude 3 Sonnet)	~38 classification calls, avg 1,800 input + 300 output tokens	~$1
AWS Lambda	~190 invocations (5 per incident: trigger, diagnose, remediate, retry, record), well within free tier	~$0
AWS Step Functions	~38 workflow executions × avg 5 state transitions	~$0
Amazon EventBridge	~38 Glue failure events/month routed to Lambda	~$0
Amazon DynamoDB	~500 reads/writes per month (failure history + remediation records)	~$0
Amazon CloudWatch	Enhanced job metrics, 8 custom alarms, log queries	~$6
Amazon SNS	~11 escalation notifications/month (29% of failures escalated)	~$0
Total		~$7/month

Common Pitfalls (Real Lessons)

Pitfall	What Happened	How We Fixed It
Bedrock classified a resource constraint as schema drift	Error message mentioned "column" in a memory OOM context; Bedrock pattern-matched on "column"	Added structured fields (exec_time_s, dpu_used) to the classification prompt; resource metrics now anchor the OOM class independently of error text
Partition repair added 847 partitions in a single call	A job with daily partitions going back 3 years had 847 missing partitions — the batch create took 4 minutes and delayed the retry	Added a max_partitions_per_repair limit of 90 (covering 3 months); older partitions flagged for manual backfill review
DPU auto-increase applied to a job that was timing out due to an infinite loop in the script	Increasing DPUs did not fix the timeout — the job just consumed more capacity before failing again	Added a check: if the same job has had a JOB_TIMEOUT in the last 7 days that was already DPU-remediated, escalate instead of re-applying the fix
Step Functions retried the job before the upstream dependency had finished	DEPENDENCY_FAILURE retry was triggered after a 15-minute wait, but the upstream job was a 40-minute process	Changed the dependency check to poll the upstream job's Glue run status every 5 minutes up to 60 minutes, rather than using a fixed wait state
Bedrock returned valid JSON wrapped in a markdown code block	The classification Lambda failed to parse the response as JSON because Bedrock wrapped it in a ```json fenced block.	Strip markdown code fences before JSON parsing; set temperature=0.1 for more deterministic output.

The DPU re-application pitfall had the most operational impact. When the self-healing system increased DPUs on a job that was timing out due to a script bug, the job ran for the full extended timeout period (3 hours) consuming 20 DPUs before failing again — generating an unexpected Glue compute charge of approximately $14 for that single job run. The recurrence guard — checking failure history before re-applying the same fix class — was added as a direct result.

Business Outcomes

Metric	Before Self-Healing	After Self-Healing	Business Impact
Pipeline failures requiring human intervention	100% (38/month)	29% (~11/month)	27 incidents per month resolved autonomously
Mean time to recovery	47 minutes average	8 minutes (auto) / 34 minutes (escalated)	Faster data availability for downstream consumers
Data engineering sprint capacity on incident response	~22%	~7%	~1.3 engineer-days per sprint recovered
Off-hours pages to engineers	~14/month	~4/month	Reduced on-call burden significantly
Demand forecasting model trained on incomplete data	2 incidents in 6 months	0 since launch	Prediction quality protected
Escalation quality	Raw error notification	Bedrock-generated briefing with root cause + suggested fix	Review time per escalation: 34 min vs. 47 min

The most operationally significant outcome was not the autonomy rate — it was the reduction in off-hours pages. Of the 38 monthly failures, approximately fourteen previously triggered off-hours alerts. After the self-healing system, that dropped to four — the genuinely complex failures that require human judgment. The data engineering team's on-call quality of life improved immediately and measurably, which is a retention and morale outcome that does not appear on a cost spreadsheet but matters to engineering leadership.

Lessons for Technology Leaders

• Root cause classification is the hard part of pipeline automation — not the remediation — Every data engineering team knows how to fix a missing partition or increase a DPU allocation. The reason they do it manually is that identifying which fix applies requires reading logs, understanding context, and applying judgment. Bedrock solves the classification problem at scale. Once classification is reliable, the remediation is straightforward to automate.
• Define the remediation tier boundary before you build — The most important architectural decision in this system is which failure classes are safe to auto-remediate and which require human judgment. Get this wrong and you either over-automate (creating hard-to-debug automated fixes on production data) or under-automate (escalating everything and building nothing useful). Spend the time defining the tier boundary with your data engineering team before writing any automation code.
• Recurrence guards are not optional — An automated fix that is applied repeatedly to the same underlying problem creates more damage than manual intervention. Every auto-remediation action must check whether the same fix has already been applied to the same failure class on the same job recently. If it has, escalate — the fix is not addressing the root cause.
• Escalation quality is half the value of the system — The self-healing system is most valuable when it fails to auto-remediate, not when it succeeds. When it escalates, it sends the data engineer a structured briefing — Bedrock's classification, the error context, the schema history, the recurrence pattern, and the suggested investigation path. That briefing reduces the escalated incident resolution time from 47 minutes to 34 minutes. The improvement comes entirely from escalation quality, not from eliminating human involvement.
• Start with one failure class, not five — We launched with partition path mismatch remediation only — the highest-frequency, lowest-risk failure class. After two weeks of validation with zero false positives, we added job timeout remediation. Then malformed record skipping. The phased approach built team trust in the system before extending its autonomy. Attempting to automate all five failure classes simultaneously would have delayed go-live by months and created multiple failure modes to debug simultaneously.

---

About the Author

Chandni Gadhvi is Program Manager – Data and AI at AeonX Digital Technology Limited, where she leads the architecture and delivery of cloud-native AI solutions for enterprise operations. She specializes in building intelligent, event-driven systems on AWS that convert operational data into business decisions. She is an advocate for data-first AI strategy and shares technical thought leadership to help engineering leaders move from pilot to production on AWS.