Modifiers

Modifiers #

Modifiers are collections of small individual tasks that manipulate data. A Modifier must be connected to a Process Step.

There has to be at least one Step to create a Modifier. A Step is a collection of Modifiers that will be run at the same time. If there are more Steps, the Modifiers that are in a particular Step will not be run until all active Modifiers in all preceding Steps finish with the SUCCESS Status.

To create a Step click the Add new step button on the right side of the list of steps in the Progress details section.

This will open a popup where you need to provide a name for the new Step.

The name must be unique for the current Process

There are three types of Modifiers: Workflows, Cleaners and Scripts. After adding a new Step, the Create modifier button will become active and you will be able to choose whether to create a Workflow, a Cleaner or a Script. When you choose to create a Workflow, the Modeler will start. When you choose to create a Cleaner or a Script, a new form will open.

Modifier Statuses #

Each Modifier can have one of the following Statuses:

  • NOT RUN YET - the Modifier has not been run yet,
  • QUEUED - the Modifier is waiting to be run, as other Modifiers are being run; only a certain number of Modifiers can be run simultaneously; that number can be modified using Configuration Variables,
  • AWAITING - set to every Modifier of the when a Process starts. It shows the user that the Modifier will run if previous step is completed successfully,
  • IN PROGRESS - the Modifier has been started but has not yet finished,
  • SUCCESS - the Modifier has finished successfully,
  • CANCELED - set to every Modifier with AWAITING status when the Process fails. It tells the user that the Modifier execution was cancelled due to one of the previous Steps failure.
  • ERROR - the Modifier has failed to finish because of an error; refer to the Modifier Run History section of each Modifier for more details.

Import Modifiers #

To import a Modifier go to the Process details section and click the Import modifier button.

This will navigate you to the Import Modifier section where you need to provide:

  1. a File with zip extension containing the exported Modifier,
  2. a Password for the file (if the exported file is password-protected; leave blank if not),
  3. the Step where to import the Modifier to,
  4. additional options:
    • whether to import the Predefined Data Sources (the exported file needed to have the Include predefined sources option checked),
    • whether to import the Predefined Data Targets (the exported file needed to have the Include predefined targets option checked),

Predefined Data Sources and Predefined Data Targets with names that are already present in ETL data_snake are ignored and not imported.

After clicking the Save button, you will return to the Process details section with the imported Modifier in the chosen step.

After clicking the Cancel button, you will return to the Process details section, without importing any Modifiers.

Modifier run logs #

Every time you run a Modifier or a Process, the Modifier Status is saved to a log entry.

To access the latest log entries, click on the small Status icon in the upper right corner of the Modifier on the list of Steps in the Process details section.

Run logs differ slightly for Workflows, Cleaners and Scripts.