PIPELINES¶

Purpose: The Pipelines module is where users build, run, and manage ETL data pipelines. A pipeline defines how data flows from a source connection to a destination connection — this page is the central hub for creating new pipelines, monitoring their status, and triggering or scheduling their execution.

Acessing Pipelines¶

Navigation: ETL Management → Pipelines

Pipelines Page (Main View)¶

This is the Pipelines listing page — the central hub where all configured pipelines are searched, monitored, and operated.

• Title: Pipelines
• Top-right corner button: ⠿ Visual Builder button (teal) — opens the drag-and-drop builder used to create a new pipeline visually.

Toolbar¶

Element	Description
Search bar	Quickly locate a pipeline by name (e.g., "Search pipelines...").
Status filter	Dropdown to filter the list by pipeline status (defaults to "All Statuses").
Sort dropdown	Orders the list by "Created Date".
Sort direction icon	Toggles ascending/descending order for the selected sort field.

Pipeline Card¶

Each configured pipeline is displayed as an individual card in the listing grid:

• Pipeline Name | The pipeline's name, shown prominently at the top-left of the card (e.g., "Mysqltest", "Clickhouse", "Mysql to ClickhouseTest").
• Status Badge | A pill in the top-right of the card indicating the pipeline's current state:
  • 🟠 DRAFT | The pipeline is configured but not yet activated/running on a schedule.
  • 🟢 ACTIVE | The pipeline is live and eligible to run.
• Source → Destination | Two labeled fields connected by an arrow, showing where data is read from and written to:
  • Source | The origin connection (e.g., "Mysql_36").
  • Destination | The target connection (e.g., "C_H", "Mysql_36").
• Last Run Info | A line showing:
  • A clock icon with the last run's date/time (e.g., "Jul 16, 2026, 05:28 PM").
  • A checkmark with the success percentage for that run (e.g., "0.00% success", "40.00% success").
• Action Buttons | A row of four buttons at the bottom of the card:

Button	Behavior
▶ Run (green outline)	Manually triggers an immediate run of the pipeline.
📅 Schedule (blue outline)	Opens scheduling options to configure the pipeline to run automatically (e.g., recurring intervals).
✏️ Edit (gray outline)	Opens the pipeline in the builder to modify its source, destination, transformations, or configuration.
🗑️ Delete (red outline)	Removes the pipeline from the system (typically with a confirmation step).

Visual Builder (Pipeline Canvas)¶

Opened via the ⠿ Visual Builder button on the Pipelines page, or the ✏️ Edit action on a pipeline card. This is a drag-and-drop canvas where a pipeline's source, transformations, and destination are assembled visually.

Header¶

Element	Description
← Back	Arrow icon at the top-left that exits the builder and returns to the Pipelines listing page.
Pipeline Name	The pipeline's title, shown next to the back arrow (e.g., "Untitled Pipeline"); editable.
Components	Toggles the left-hand Components palette open/closed.
Settings	Opens pipeline-level configuration (e.g., name, description, general options).
CRM Allocation	Opens allocation/routing settings tying the pipeline's output to CRM records or teams.
⊖ / ⊕ Zoom Controls	Icons to zoom the canvas out/in, alongside a live zoom percentage (e.g., "91%").
Fit-to-Screen	Resets the canvas view to fit all placed components on screen.
Minimap toggle	Icon to show/hide a small overview map of the canvas.
💾 Save (blue)	Saves the current pipeline configuration as a draft.
▶ Run (green)	Executes the pipeline immediately using its current configuration.
📤 Publish (teal)	Activates the pipeline, making it live/schedulable.
⏸ Pause	Pauses a currently active/running pipeline.

Left Sidebar — Components Palette¶

A searchable palette of draggable components, grouped by category:

• Search bar | Filters the component list by name.

Sources¶

Components that extract data into the pipeline: - PostgreSQL Source, MySQL Source, ClickHouse Source, MongoDB Source, Dataset, Webhook, Kafka Source

Transformations¶

Components that process, clean, or reshape data mid-pipeline: - Map Columns, Filter Rows, Join, Remove Duplicates, Null Handling, Aggregate, Validate Schema, Vehicle Service Forecast, Data Drop

Destinations¶

Components that load the processed data into a target system, or trigger a downstream action:

• BigQuery Destination, PostgreSQL Destination, MySQL Destination, ClickHouse Destination, MongoDB Destination
• Preview Data | Inspects the pipeline's output data without writing it anywhere.
• Respond to Webhook | Sends the processed data back as a webhook response.
• Master Form Upsert | Inserts or updates records in a master form/dataset.
• Workflow Allocation | Routes the processed data into a workflow allocation/assignment step.

Canvas (Empty State)¶

When a pipeline has no components placed yet, the canvas shows a guided empty state:

🎨 Start Building Your Pipeline "Drag components from the left palette onto the canvas"

1️⃣ Add a Source to extract data 2️⃣ Add Transformations to process data 3️⃣ Add a Destination to load data

Canvas Controls (Bottom-Right)¶

Control	Description
+	Zoom in on the canvas.
–	Zoom out on the canvas.
⬜ Fit to Screen	Resets zoom/pan to fit the whole pipeline in view.
Toggle Grid	Toggles the grid background on/off, helping with component alignment.

Adding and Configuring a Source Node¶

Step 1: Add a Source Node¶

Select the desired source from the Source menu, then drag and drop it onto the canvas.

Step 2: Configure the Source Node¶

Double-click the source node on the canvas to open its configuration panel. Here, the user can:

• Enter a Node Name (required) and an optional Description
• Under Select Dataset, choose the dataset to use as the data source for this node
• Under Select Columns, choose which columns to include (a counter shows how many of the total columns are selected)
• Optionally add Data Filters to define conditions. Click + Add Filter to create a filter rule, which includes:
  • Select field — choose the column to filter on
  • Operator — choose a condition (e.g., Equals (=))
  • Enter value — provide the value to match against
  • Use the trash icon to remove a filter rule; click + Add Filter again to add more
• Optionally set a Record Limit to cap the number of records fetched (leave empty to fetch all)

Click Save to apply the configuration, or Delete / Cancel to remove the node or discard changes.

Note: At least one column must be selected before saving — the panel shows a warning ("Please select at least one column from the dataset") until this is satisfied.

Step 3: Configure CRM Allocation¶

Note: The dataset should be uploaded and selected (Steps 1–2) before configuring CRM Allocation.

Click the CRM Allocation node to open its configuration panel, then check Enable CRM Allocation. When enabled, records loaded by this pipeline will be automatically allocated to CRM workflow stages.

Once enabled, the following options become available:

• Route through Master Form — when enabled, ETL data is first inserted/updated in a Master Form, then auto-allocated to workflow stages via the master form's allocation rules
• Routing Rules — define rules to route records to different workflows based on field values. Rules are evaluated in priority order (lower number = higher priority). Click + Add Routing Rule to create one
• Default Target — set the Workflow and Stage that records will be allocated to if they don't match any routing rule
• Field Mappings — map pipeline output columns to CRM form fields so mapped fields are pre-filled in the form. Click + Add Mapping to create one
• Additional Settings:
  • Batch Size — number of records per Kafka batch (1–1000)
  • Allocation Limit (optional) — maximum number of records to allocate (leave empty for no limit)
  • Priority — set the allocation priority
• Deduplication — control whether previously allocated records are re-processed:
  • Skip records already allocated to this stage
  • Skip records already allocated to any stage
  • Force allocate (ignore all dedup checks)
• User Assignment Method — choose how users are assigned to allocated records (e.g., Static: Fixed User for All Records)
• User Filters (optional) — narrow down the available user list (all filters are applied as an intersection/AND):
  • System Mapping — filter users by system definition mapping
  • Role Filter — only users with selected roles will be available
• Filter Users by Hierarchy (optional) — filter the user list by hierarchy node
• Assign To User — assign the same user to all records in this pipeline (defaults to stage config if not set)
• Test Allocation — test routing rules and field mappings with sample data before running the pipeline. Enter sample JSON data and click Run Test

Click Save CRM Config to apply the configuration.

Step 4: Run the Pipeline¶

Once the dataset is Uploaded and the CRM Allocation is configured, click Run. The data will be allocated based on the configuration set in CRM Allocation (routing rules, default target, field mappings, etc.), and the allocated data will then be visible in the Master Form.

Step 5: Save and Publish the Pipeline¶

Click Save to save the pipeline configuration, and then click Publish to make the pipeline active.Once published the pipeline will be active.

Saved Pipelines List¶

Once all pipelines are created and saved, they are visible in card format on the Pipelines landing page. Each card shows:

• The pipeline name and a status badge (e.g., DRAFT)
• Source → Destination — the data flow direction for that pipeline
• The last run timestamp and success rate (e.g., "Jul 20, 2026, 11:06 AM · 100.00% success")
• Action buttons: Run, Schedule, Edit, and Delete

Run¶

The Run button appears on the pipeline card, but it cannot be used until the pipeline is opened — the pipeline must be run from within the canvas.

Schedule the Pipeline (Main Page)¶

Once published, a pipeline can be run on a scheduled basis using the Schedule option. On the pipeline card, click the Schedule button to open the Configure Pipeline Schedule tab. Here, the user can choose a Schedule Type:

• Manual — run only when manually triggered
• At Specific Time — run daily, weekly, or monthly at exact times
• Repeat Continuously — run every X minutes/hours

Additional options are available under Show Advanced Settings. A Schedule Summary at the bottom reflects the current selection (e.g., "Pipeline will run only when manually triggered").

Click Save Schedule to apply, or Cancel to discard changes.

Edit¶

Clicking the Edit button on the card opens the pipeline in the canvas, where the user can modify its nodes and configuration (source, CRM allocation, etc.).

Delete¶

Clicking the Delete button on the card permanently removes the pipeline from the list. This action cannot be undone, so a confirmation prompt is typically shown before the pipeline is deleted.

Note: Once the pipeline is executed, detailed execution information can be checked in the Jobs section.

Frequently Asked Questions (FAQ)¶

Q: What is a pipeline in Advaita Data Bridge?

A: A pipeline defines how data flows from a source connection to a destination connection, including any transformations applied along the way. It's the core unit of work in the ETL module.

Q: What's the difference between DRAFT and ACTIVE status?

A: DRAFT means the pipeline is configured but hasn't been published — it won't run on a schedule. ACTIVE means the pipeline has been published and is live/eligible to run, whether manually or on a schedule.

Q: Why can't I run a pipeline directly from its card on the listing page?

A: The Run button on the card requires the pipeline to be opened in the canvas first — pipelines are executed from within the Visual Builder, not directly from the listing view.

Q: What's the difference between Save and Publish in the Visual Builder?

A: Save stores your current configuration as a draft, so you can come back and continue editing later. Publish activates the pipeline, making it live and schedulable. You typically save first, then publish once you're ready to go live.

Q: Do I need to configure CRM Allocation for every pipeline?

A: No — CRM Allocation is optional. It's only needed if you want records loaded by the pipeline to be automatically routed into CRM workflow stages. If you just need raw data movement between systems, you can skip this step.

Q: What happens if a record doesn't match any Routing Rule?

A: It falls back to the Default Target — the Workflow and Stage you've configured to catch unmatched records.

Q: Why is my Source node configuration not saving?

A: At least one column must be selected under Select Columns before saving. If none are selected, the panel will show a warning and block the save.

Q: What do the Deduplication options do?

A: They control whether records that have already been allocated get processed again:

• Skip if allocated to this stage — avoids duplicates within the same stage
• Skip if allocated to any stage — avoids duplicates across all stages
• Force allocate — ignores dedup checks entirely and allocates regardless

Q: What's the difference between "Manual", "At Specific Time", and "Repeat Continuously" schedule types?

A: - Manual — the pipeline only runs when someone triggers it by hand - At Specific Time — runs on a fixed daily/weekly/monthly schedule - Repeat Continuously — runs on a recurring interval (every X minutes/hours)

Q: Where can I see the results of a pipeline run?

A: Detailed execution information for each run is available in the Jobs section, which tracks succeeded, failed, and running executions.

Q: Can I undo a pipeline deletion?

A: No — deleting a pipeline is permanent. A confirmation prompt is shown before deletion to help prevent accidental removal.

Q: What's the Batch Size setting used for?

A: It controls how many records are grouped per Kafka batch during CRM allocation, with a range of 1–1000.

Q: How do I test my routing rules before running the full pipeline?

A: Use the Test Allocation feature in the CRM Allocation panel — enter sample JSON data and click Run Test to preview how records would be routed without actually running the pipeline.