Transform Map

ETL Transform Map defines how extracted data is transformed and loaded into target DocTypes. It includes field mappings, value transformations, business logic scripts, and upsert rules.

Field Reference

Basic Configuration

Field	Type	Required	Description
Transform Name	Data	Yes	Unique identifier for this transform map
Target DocType	Link	Yes	Frappe DocType to create/update records in
Application	Link	Yes	My Application this transform belongs to
Active	Check	No	Enable/disable this transform (default: unchecked)

Operation Control

Field	Type	Required	Description
Allow Insert	Check	No	Allow creation of new records
Allow Update	Check	No	Allow updating existing records
Skip Business Logic	Check	No	Bypass DocType controllers and triggers
Update Modified Fields	Check	No	Update modified timestamps (when Skip Business Logic enabled)

Note: Either Allow Insert or Allow Update must be enabled.

Field Mapping

The Field Map table defines how source fields map to target DocType fields:

Field	Type	Required	Description
Source Path	Data	Yes	Source field name or JSONPath expression
Target Field	Data	Yes	Target DocType field name
Coalesce	Check	No	Use this field for record matching (upsert)
Default Value	Data	No	Fallback value if source field is empty

Value Mapping

The Value Map table transforms source values to target values:

Field	Type	Required	Description
Source Field	Data	Yes	Source field to check
Source Value	Data	Yes	Source value to match
Target Field	Data	Yes	Target field to set
Target Value	Data	Yes	Value to set in target field

Transform Scripts

Transform scripts are Python code snippets that run at different stages of the transformation process. All scripts have access to the result object for returning values and setting flags.

On Start Script

Purpose: Runs once before the entire transformation process begins.

Available Variables: - transform_map (dict): Complete transform map configuration - import_batch (dict): Import batch details

Script Capabilities: - Set result['skip'] = True to skip entire transformation - Initialize counters or setup data structures - Validate batch before processing

Example:

# Skip transformation if too many records
if import_batch['rows'] > 10000:
    result['skip'] = True
    result['skip_reason'] = 'Batch too large for processing'

mantera.log_info("ETL", f"Starting transform for {import_batch['rows']} rows")

On Before Script

Purpose: Runs before each row is transformed.

Available Variables: - source_record (dict): Raw source data for current row - result (dict): Mutable result object

Script Capabilities: - Set result['skip'] = True to skip current row - Add/modify fields via result['set_values'] - Apply conditional logic and data cleansing

Example:

# Skip rows with invalid email
if not source_record.get('email') or '@' not in source_record.get('email', ''):
    result['skip'] = True
    result['skip_reason'] = 'Invalid email address'
else:
    # Set values for field mapping
    result['set_values'] = {}
    result['set_values']['status'] = 'Active' if source_record.get('active') else 'Inactive'
    result['set_values']['full_name'] = f"{source_record.get('first_name', '')} {source_record.get('last_name', '')}"

On After Script

Purpose: Runs after each row is successfully transformed and saved.

Available Variables: - source_record (dict): Original source data - doc_values (dict): Final values that were saved - doc_name (str): Name of created/updated document - action (str): "insert", "update", or "skip"

Example:

# Log successful operations
if action == "insert":
    mantera.log_info("ETL Success", f"Created new customer: {doc_name}")
elif action == "update":
    mantera.log_info("ETL Success", f"Updated customer: {doc_name}")

On Complete Script

Purpose: Runs once after the entire transformation process completes.

Available Variables: - stats (dict): Final statistics with keys: inserts, updates, skipped, errors

Example:

total_processed = stats['inserts'] + stats['updates']
result['summary'] = f"Processed {total_processed} records, {stats['errors']} errors"

# Send notification for high error rates
if stats['errors'] > total_processed * 0.1:
    frappe.sendmail(
        recipients=['admin@company.com'],
        subject='ETL High Error Rate',
        message=f"ETL job completed with {stats['errors']} errors out of {total_processed + stats['errors']} records"
    )

Upsert Logic

Records are matched for updates using Coalesce fields from the Field Map:

Insert: If no matching record found and Allow Insert enabled
Update: If matching record found and Allow Update enabled
Skip: If no operation is allowed or both flags disabled

Actions

Run Transform

Launches a dialog to select an ETL Import Batch for transformation. Creates an ETL Transform Run with detailed event logging.

Usage Examples

Customer Import Example

Transform Name: Customer Data Import
Target DocType: Customer
Allow Insert: ✓
Allow Update: ✓

Field Map:
- Source: customer_id → Target: customer_code (Coalesce: ✓)
- Source: company_name → Target: customer_name
- Source: email → Target: email_id
- Source: phone → Target: mobile_no

Value Map:
- Source Field: status, Source Value: A → Target Field: disabled, Target Value: 0
- Source Field: status, Source Value: I → Target Field: disabled, Target Value: 1

Script Validation

Transform scripts are validated to ensure security: - No import statements allowed - No def function definitions - No return statements
- Access only to approved script methods - Must use result object for output ```