Skip to main content

Varicent ELT Assistant

Databricks

Connect your Varicent ELT and Databricks account to import your data.

  1. In Varicent ELT, select the Data module.

  2. Click the Import data tab.

  3. In the third-party sources section, click Databricks.

  4. Click New connection.

  5. On the Connect to your Model page, under Connector name, enter a unique connection name.

  6. Under Workspace URL, enter the URL of your Databricks workspace. For example, https://<workspace-name>.cloud.databricks.com.

  7. Under SQL Warehouse, enter the ID of the SQL warehouse used to run queries. You can obtain this value from the SQL warehouse details in Databricks.

  8. Under Access token, enter the Databricks access token used to authenticate the connection.

  9. Click Next.

  10. On the Choose data sets page, the available data sets are shown. Select which data sets you want to include, or select Include all, and then click Next.

  11. On the Import data page, review your data and select the data to obfuscate (if any), select Keep out of bound dates to maintain any dates outside the date range, and then click Next.

  12. On the Filter data page, optionally click Filter data to refine your data, or click Import data to continue.

    1. Filter the rows in your data by selecting the column to filter, the filter category and the value or text.

    2. Click Import data to continue.

    Note

    For more information about how to configure data filters, see: Filter data.

  13. On the Review page, review the data sets that you are importing, and then click Import data.

    You are redirected to the My data page where you can review the progress of the import. After the data is imported, you can click + Create pipe.

Import new records incrementally

If a Databricks table contains a date or timestamp column that identifies when records were added, you can configure a filter that retrieves records after a specified date or timestamp.

Use this configuration with a scheduler to perform recurring imports of new records.

Important

Varicent ELT does not calculate the difference between the current and previous contents of the Databricks table. The source table must contain a suitable date or timestamp column, and the import filter must identify the records to retrieve.

For information about running imports automatically, see Schedule data.

Error handling

The following errors may appear during the import process:

Table 12. 

Error

Possible cause

Resolution

The connection cannot be established.

The workspace URL, warehouse ID, or access token is incorrect.

Verify the connection values in Databricks and edit the connection.

Authentication failure.

The access token is invalid, expired, or revoked.

Generate or obtain a valid token and update the connection.

A catalog, schema, or table is not listed.

The Databricks identity does not have permission to access it.

Ask a Databricks administrator to grant the required permissions on the catalog, schema, or table.

The SQL warehouse cannot be accessed.

The warehouse ID is incorrect, the warehouse is unavailable, or the identity cannot use it.

Verify the warehouse ID and ensure that the identity has permission to use the SQL warehouse.

A large import times out or fails.

The selected table contains more data than can be retrieved within the available processing time.

Apply a source filter to reduce the number of records and retry the import.

An incremental import retrieves unexpected records.

The selected date filter does not accurately identify new records.

Review the source date or timestamp values and update the filter condition.

The table schema changed after the data source was created.

Columns were added, removed, renamed, or assigned different data types in Databricks.

Review the source table and refresh or reconfigure the import.



Connection creation, connection updates, import runs, and import failures are recorded in the Varicent ELT audit log.