Transform Run
ETL Transform Run records the execution of an ETL Transform Map against an ETL Import Batch. It tracks transformation statistics and provides detailed event logs for monitoring and troubleshooting.
Note: This DocType is system-generated when running transformations. Transform runs cannot be created manually.
Field Reference
Run Information
| Field | Type | Description |
|---|---|---|
| Name | Auto | System-generated identifier (TRUN-YYYY-MM-DD-#####) |
| Transform Map | Link | ETL Transform Map used for transformation |
| Import Batch | Link | ETL Import Batch that was processed |
| Status | Select | Current run status (Running, Complete, Failed, Skipped) |
| Summary | Small Text | Brief description of results or errors |
Performance Metrics
| Field | Type | Description |
|---|---|---|
| Duration MS | Int | Total execution time in milliseconds |
| Started At | Datetime | When transformation began |
| Finished At | Datetime | When transformation completed |
Operation Counts
| Field | Type | Description |
|---|---|---|
| Inserts | Int | Number of new records created |
| Updates | Int | Number of existing records updated |
| Skipped | Int | Number of records skipped (no operation performed) |
| Errors | Int | Number of records that failed processing |
Event Logging
| Field | Type | Description |
|---|---|---|
| Transform Event | Table | Detailed log of processing events (child table) |
Status Values
| Status | Description |
|---|---|
| Running | Transformation currently in progress |
| Complete | Transformation finished successfully (errors may still exist) |
| Failed | Transformation stopped due to critical error |
| Skipped | Transformation skipped by On Start script |
Transform Events
Each Transform Event child record logs individual processing actions:
| Field | Type | Description |
|---|---|---|
| Level | Select | Event severity (INFO, WARN, ERROR) |
| Chunk Ref | Link | ETL Import Chunk being processed |
| Row Index | Int | Position within the chunk (1-based) |
| Target Document | Data | Name of created/updated target record |
| Message | Small Text | Description of what occurred |
Event Types
INFO Events: - Successful record creation: "Successfully created Customer: CUST-001" - Successful record update: "Successfully updated Customer: CUST-001"
WARN Events: - Script-based skips: "Row skipped: Missing required email field" - No field mappings: "No values mapped from source record" - Operation restrictions: "Record skipped - no insert/update allowed"
ERROR Events: - JSON parsing failures: "Invalid JSON in source data" - Processing exceptions: "Processing error: ValidationError in Customer creation" - Chunk-level errors: "Chunk processing error: Database connection lost"
Monitoring Transforms
Real-time Monitoring
During transformation execution: - Status shows "Running" - Counters update as records are processed - Events are logged for each row operation - Duration tracks elapsed processing time
Post-Execution Analysis
After transformation completion: - Review final counts (inserts/updates/skipped/errors) - Filter Transform Events by level to focus on issues - Calculate success rate: (inserts + updates) / (total records - skipped) - Compare duration against batch size for performance analysis
Performance Analysis
Key Metrics
- Throughput: Records processed per second
- Error Rate: Errors / total records processed
- Skip Rate: Skipped / total records processed
- Success Rate: (Inserts + Updates) / total records processed
Optimization Tips
- High error rates may indicate data quality issues
- High skip rates may indicate overly restrictive transform logic
- Long durations may indicate chunking size issues
- Review Transform Events for specific bottlenecks
Troubleshooting
Common Issues
High Error Rates: 1. Review ERROR level Transform Events 2. Check source data quality and formats 3. Verify target DocType field requirements 4. Ensure proper field mapping configuration
Poor Performance: 1. Reduce chunk size for memory-intensive transforms 2. Optimize On Before/On After scripts 3. Check database indexing on coalesce fields 4. Review transform logic complexity
Unexpected Skips:
1. Review WARN level Transform Events
2. Check Allow Insert/Allow Update settings
3. Verify coalesce field matching logic
4. Review On Before script skip conditions
Related DocTypes
- ETL Transform Map: Configuration used for this run
- ETL Import Batch: Source data that was processed
- ETL Transform Event: Detailed event logs (child records)