Skip to main content

Varicent ELT Help Center

Pipe

Abstract

Bring pipe logic from another pipe into the current pipe.

Bring pipe logic from another pipe into the current pipe. You can connect two or more data sources of your choice into the Pipe tool. The Pipe tool automatically maps the columns names after you connect the data sources.

Use the Pipe tool in both the Standard and Big pipes. Copy the pipe that you want to bring into a new pipe from the Pipe page, and use the Pipe tool to bring it in. For more information about copying a pipe, see: Copy a pipe.

Note

You cannot bring data from a Standard pipe into a Big pipe and you cannot bring data from a Big pipe into a Standard pipe. For example, in the Standard pipe, you will only see Standard pipe options available in the pipe configuration drop-down menu.

Input and output

The number of inputs feeding into the Pipe tool must be the same, and you can bring in any data sources of your choice. Ensure that you connect all of your data sources in the Pipe tool. Automatic column mapping is based on column names.

If the column names or column types are different, you must manually map the columns in a provided form.

When to use this tool

Use when you want to bring in data from another pipe or re-use a portion of pipe logic in more than one workbench.

Note

When building pipes, anything connected by a Pipe tool will be built together.

Configuring the Pipe tool

  1. Click + Tool on the toolbar.

  2. Type or search for Pipe tool. Click + Add tool.

  3. In the configuration pane, click the Pipe name drop-down menu.

  4. Select the pipe that you want to use. In the Pipe name field, you can preview the pipe or open the pipe in a new window.

  5. In the Pipe inputs field, click Connect to input for each node.

    Important

    By default, the nodes are not automatically connected. You must connect each input.

  6. In the Configure connection pane, configure the following fields:

    1. In the Source drop-down menu, select the tool source from the pipe that you want to use.

    2. In the Target field, select the target for the input.

  7. In the Map columns to target field, use the toggle in the off position to display all columns, including unmapped columns, or turn the toggle on to conceal the mapped columns.

  8. Click Done.

Usage example

Let's say you want to bring an output from pipe "Function" into your current pipe, "Sandbox". There are three things you need to make sure of:

  1. "Function" begins with a placeholder data set, and ends with an Export tool.

  2. Pipes "Function" and "Sandbox" have the same number of inputs and outputs. If there is one input to the Pipe tool in "Sandbox", there should be one placeholder data set in "Function".

  3. The placeholder data sets in "Function" share the same column names and types as the inputs to the Pipe tool in "Sandbox".

Pipe tool upgrade

If you are using an older version of the pipe tool, you may be prompted to upgrade to the latest version. Upgrading can enhance your experience and unlock new features.

However, there is a warning when you are prompted to upgrade. Some of your existing configurations may not transfer over to the latest pipe tool version. These configurations could be lost in the process.

  • If you want to move to the newest version, you can click Continue to proceed with the switch to the new Pipe tool.

  • If you don't want to continue and potentially lose some of your pipe tool configurations, you can click Cancel.