Delta Extraction Configuration

Delta extraction enables incremental data updates, reducing processing time and system load by extracting only changed records since the last update.

General Prerequisites

• Source System Requirements

  • Timestamp Fields: Modified date, created date, or last updated fields

  • Change Tracking: System-level audit logs or change documents

  • Incremental Keys: Sequence numbers or version fields

  • API Support: Delta-capable endpoints for API-based sources

• JustPerform Prerequisites

  • Initial Full Load: Complete data extraction to establish baseline

  • Watermark Management: System tracking of last extraction point

  • Error Handling: Recovery mechanisms for failed delta extractions

  • Monitoring: Tracking delta volume and extraction success rates

Implementation Guidelines

1. Identify the Delta Fields:

   Field type	Parameters
   Standard Fields	LastModifiedDate LastUpdatedTimestamp ModifiedOn ChangedAt
   System-Specific Fields	SAP: ERSDA ERDAT AEDAT Salesforce: LastModifiedDate SystemModstamp Dynamics: modifiedon createdon

2. Configure Delta Rules

   • Time Window: Define extraction frequency (hourly, daily, etc.)

   • Overlap Buffer: Include small time buffer to avoid missing records

   • Maximum Delta Size: Set limits to prevent oversized extractions

   • Fallback Strategy: Switch to full load if delta size exceeds threshold

3. Handle Delta Challenges

   • Clock Synchronization: Account for time zone differences

   • Record Deletions: Implement soft delete tracking or separate deletion logs  

   • Late Arriving Data: Configure appropriate lookback windows

   • Schema Changes: Monitor for structural modifications

4. Monitor Delta Performance

   • Extraction Volume: Track record counts per delta run

   • Processing Time: Monitor extraction duration trends

   • Error Rates: Identify and resolve delta-specific issues

   • Data Quality: Validate delta completeness and accuracy