Added endpoint that calculates all parameters for process.
Added toggle use_cached_parameters for modeler views. Default is switched on and views
use cache parameters. If switched off then endpoints refresh parameters values.
Changed properties field on ModifierRunLog model to execution_state that stores the information
about the modifier’s fields at the moment of the execution.
Process execution state is now stored in process run logs.
Process execution state holds the information about steps’ details, inactive modifiers’ details,
parameters and the status of the process and its inactive modifiers.
Process execution state can be joined with modifiers’ execution states from related
ModifierRunLogs to also include the information about the active modifiers.
Added REST endpoint that gives the state of the last process execution.
The endpoint supports authentication by process token.
The historical process executions can be accessed by passing task_id query parameter.
The endpoint returns the HTTP 409 Conflict response when the process was never run.
From now on, changing the process or its components while it is running will not affect current
process execution.
Fixed new lines not being preserved in python exception traceback due to parameter error when
saving or evaluation on parameter create/update form.
Moved parameter management view tests to a separate file, enhanced them, added missing tests for
them.
Added a test for parameter list REST view.
Improved test for lookup_set_value function.
Added default ordering for ProcessParameter and LicenseKey models.
Fixed some typos.
Changed handling of simple parameter’s invalid type on management views — now there’s a short
concrete error message instead of a generic one and needless traceback.
Added validation fields (is_valid, last_validated) on Modifier model. They hold state of validation.
Added endpoints for refreshing validation state for entire process or single modifier.
Added information about validation status on process details view.
Optimized run process action. Now parameters are only calculated in celery tasks. There is no
redundant parameters’ calculation in validation stage before the celery task.
Added export and import of process parameters on backend and frontend. They can be included in
the workflow when exporting it and then imported along the same way predefined sources and
targets are.
Code from script components is now executed inside the function.
ScriptSource and ScriptTransformation must now return the pandas.DataFrame object.
All existing scripts will be automatically changed to return old OUTPUT_DATA variable.
ScriptTarget and ScriptModifier do not need to return anything.
In modeler views computing parameters added returning computed parameters (if available) in
ETL-PROCESS-PARAMETERS HTTP header. Removed parameters from returned json (which was absent in
case of an error anyway) in those views.
Improve errors for missing parameters and invalid parameter lookups on modeler. E.g. now missing
parameter errors are marked on individual nodes in their error boxes.
Moved parameter computation and substitution from EtlGraphAllSerializer to a validator in
EtlGraph. This fixes parameters being processed for nodes not in the current graph in case of
subgraph validation.
Add structure description backend. Structure description is short description about bound resource
to connector e.g. database table, dataflow in Fusion Registry etc. It works for all databases
and Fusion Registry (sources and targets).
Run logs statuses are now displayed based on celery task results.
status field on RunLog model is now a property. Added final_status database field to hold
the final state of the task.
Added task_id field to RunLogs model that holds the id of a celery task that corresponds to
the RunLog.
Added RunLogs cleaning on Celery workers startup:
Cleaning is performed only if no other worker is consuming from the queue of the worker
starting up. If two workers start simultaneously, only one performs the cleaning.
Cleaning takes all RunLogs with task_id but without final_status, and checks if their
tasks are present on the queues.
For ModifierRunLogs, if task is in PENDING state, it also checks if Process task exists.
New tasks cannot be added to the queue when cleaning is happening.
Added periodic task that checks for tasks lost due to worker crash:
The task checks RunLogs with STARTED status in celery and checks if the worker that was
handling the task is still alive.
Task process_run now checks worker health while waiting for run_modifier task to end. It is
done the same way as described in the point above.
Added 2 new statuses for Modifiers - AWAITING and CANCELED:
AWAITING status is set to every modifier of the process when it starts. It shows the user that
the modifier will run if previous step is completed successfully. Does not appear in modifiers
ran solo.
CANCELED status is set to every modifier with AWAITING status when the process fails. It
tells the user that the modifier execution was canceled due to one of the previous steps failure.
Does not appear in modifiers ran solo.
Changed prefixes in all celery-related environmental variables in settings from ETL_CELERY_ to
CELERY_.
Removal test/edit button from source/target modals if they don’t contain extra fields. In such cases, properties validation is triggered after submitting the form.
Changed all filters layout to be in one line, change style of all inputs, selects etc. to more modern,
add datetimepicker to edit api token in process details.
Cleaner allows to clear the database table of one of
the predefined targets,
Currently, Cleaner support only DELETE clearing method,
Tables are cleared in specified order to ensure for example that tables with RESTRICT
relations will be cleared properly,
Cleaners check the databases for connection and table
existence to lower the chance of mid-run failure where some databases could be cleared, and
some do not,
On error, run logs shows which databases have been cleared and which did not.
Cleaner can be exported, imported, copied and deleted
similarly to Workflows,
Added form that allows to create or modify a Cleaner,
Process details:
Changed Create workflow button to Create modifier dropdown button that allows to choose
between Workflow and
Cleaner,
Changed Import workflow button to Import modifier button,
Modifier blocks now have a colored backgrounds based on types.
Workflows are green while
Cleaners are lavender. They also have their icons displayed in the background like the watermark,
Changed workflow import and export urls (renamed workflow to modifier),
In exported zip file, changed json file name that contains modifier from workflow.json to
modifier.json. Files containing old file name can be imported still,
Added ‘use_predef_properties’ for graph node fields. Variable defines whether use properties from
predefined object in database or from node properties.
Added endpoint for function field data for non predef objects.
Added endpoint for connector properties validation (serializer validation + test connection).
added driver field (char) with choices. It holds the name of a class that defines the
structure of properties and action that is performed on Modifier execution.
Currently, the only choice is Modifier.
Renamed the modifier to modifier in most of the places in the code. In some places that
relates exactly to modifiers it was left unchanged, ex. ModifierViewSet, import, export,
modeler.
Foreign keys and related managers in other models changed their names to match new model name.
Renamed ModifierRunLogs model to ModifierRunLogs:
Foreign keys and related managers in other models changed their names to match new model name.
Renamed ModifierStep model to ProcessStep.
Foreign keys and related managers in other models changed their names to match new model name.
Adapted front-end to new endpoint names and parameters.
In some places in the application interface renamed Modifier to Modifier:
Changed labels on modifier description from Modifier to Modifier.
Renamed Modifier name column in process error table to Modifier name.
Added Modifier type column in process error table. It holds the name of Modifier’s driver.
On process details, changed Modifiers header to Execution plan above the window with steps.
Added labels with tooltips next to truthy/falsy values in ConvertModal during casting on boolean type (Truthy, Falsy, Example, ?).
Fixed initialization of select2 selects in truthy/falsy values and added default (special) values in dropdown (Empty string, None).
If we enter the value manually, we will get information that it should be selected from the dropdown list.
(Special values should be chosen from dropdown.)
Fixed select2 values matching. From now e.g. adding substrings and confirm values by pressing enter works.
Added e-mail notifications that are sent to users with administration permission when any
process or standalone modifier execution fails.
Added “Mailing” tab in app settings section that contains form that allows to change SMTP server
connection settings. Available only to users with administration permission.
Added Constance config section in django admin that allows to change any configuration that has
been registered in Constance. Available only to superusers.
Fixed disappearing password after testing connection on predefined connector create view.