Databricks export
Connect Varicent ELT to Databricks to export the output of a Varicent ELT pipe to an existing Databricks table.
The target table must exist before you run the export. You can append rows to the table or overwrite its existing data.
Before you begin
Complete the following tasks before configuring an export:
Create the target table in Databricks.
Confirm the target catalog, schema, and table name.
Ensure that the Databricks identity used by the connection can use the SQL warehouse and write to the target table.
Review the source and target columns.
The Databricks export writes columns that match columns in the existing target table. Source columns without a matching target column are not written to the table.
Exporting
The Databricks connection is shared by the import and export connectors. If you already created a Databricks import connection, you can use the same connection for an export.
In Varicent ELT, go to the module.
Open the pipe that contains the data you want to export.
Add an Export tool to the pipe.
Connect the data-producing tool to the Export tool.
In the Export tool configuration, open the Connection list.
Click .
Select Databricks.
Enter a unique name for the connection.
Under Workspace URL, enter the URL of your Databricks workspace.
Under SQL warehouse ID, enter the ID of the SQL warehouse used to run the export.
Under Access token, enter the Databricks access token used to authenticate the connection.
Click or .
After the connection is created, you are returned to the Export tool configuration.
Configuring a Databricks export
In Varicent ELT, go to the module.
Open the pipe that contains the data you want to export.
Add or select the Export tool.
From the Connection list, select your Databricks connection.
Configure the target table:
Under Catalog, enter the Databricks catalog that contains the target table.
Under Schema, enter the schema that contains the target table.
Under Table name, enter the name of the existing Databricks table.
Under Write mode, select one of the following write modes:
Append: Adds the exported rows to the existing rows in the target table. Existing rows are not removed.
Overwrite: Replaces the existing data in the target table with the exported data.
Review the source and target table columns.
Click to run the export immediately, or configure a schedule.
In the Create an export pane, enter an optional export name and review the input and output.
Confirm the export.
The export appears in the export log.
Caution
The Overwrite mode replaces the existing data in the target table. Confirm that the correct catalog, schema, and table are selected before running the export.
Column matching tables
The target table must already exist in Databricks. During an export, Varicent ELT writes source columns that match columns in the target table. The export does not fail solely because the source contains additional columns not present in the target table.
Before running the export, verify the following:
Required target columns have matching source columns.
Matching columns use compatible data types.
The Databricks identity can write to the target table.
The selected write mode is appropriate for the target data.
Using an existing Databricks connection
In Varicent ELT, go to the Pipes module.
Open the pipe that contains the data to export.
Add or select the Export tool.
Open the Connection list.
Search for and select the existing Databricks connection.
Select the target catalog and schema.
Enter or select the existing target table.
Select Append or Overwrite.
Click Export now, or configure a schedule.
For more information about schedules, see Schedule data.
Error handling
The following errors may appear during the export process:
Issue | Possible cause | Resolution |
The connection cannot be established. | The workspace URL, warehouse ID, or access token is incorrect. | Verify the connection values and update the connection. |
Authentication fails. | The access token is invalid, expired, or revoked. | Generate or obtain a valid token and update the connection. |
The target catalog or schema is not available. | The Databricks identity cannot access it. | Ask a Databricks administrator to grant access to the catalog and schema. |
The target table cannot be found. | The table does not exist, or the catalog, schema, or table name is incorrect. | Create the table in Databricks or correct the target configuration. |
The export fails with a permission error. | The Databricks identity does not have permission to write to the table or use the SQL warehouse. | Grant the required warehouse and table privileges. |
The export fails because of a schema or data-type error. | A source value is incompatible with the corresponding target column. | Compare the source and target schemas and convert the source columns to compatible data types. |
Some source columns are not exported. | The source columns do not have matching columns in the target table. | Add the required columns to the target table or rename the source columns before the Export tool. |
Duplicate rows appear after an export. | Append was selected and the exported rows were already present. | Remove the duplicate rows in Databricks or use an appropriate overwrite or deduplication process. |
Existing data was replaced unexpectedly. | The export used Overwrite mode. | Restore the data from Databricks recovery or backup facilities, if available, and review the export configuration. |
The export times out. | The SQL warehouse is unavailable or the export volume exceeds the available processing time. | Verify that the warehouse is running and retry the export. If necessary, export a smaller data set. |
Connection creation, connection updates, export runs, and export failures are recorded in the Varicent ELT audit log.