Triggering Taskflow based on a File Event using File Listener

1. Introduction

A taskflow can be invoked based on a file event (or) you can plan a taskflow execution through file events in Informatica Cloud.

So, What is a File Event?
The availability of new files or deletion or updating existing files in a specific defined directory in a defined pattern is called as a File Event.

To capture a file event and trigger tasks based on that event, Informatica Cloud provides a component called File Listener.

2. File Listener in Informatica Cloud

A File Listener listens to files on a defined location. You define a file listener that listens to a specific folder and the file pattern. A file event is detected when new files arrive to the monitored folder or the files in the monitored folder are updated or deleted.

You can define the file events that trigger a notification to the following assets:

• Taskflows
• File ingestion tasks
• B2B Gateway partners

These assets use file listeners to monitor specific folders, and receive notification through a call-back API when a file event occurs. 

Connector File Listener log entries are listed on the File Transfer Logs page in Monitor.

3. Configuring a File Listener

Perform the following steps to configure a file listener:

3.1 Create a File Listener

Login to Data Integration Services, click New > Components > File Listener, and then click Create to create a File Listener.

3.2 Configure File Listener details

Configure the following File Listener details

• File Listener Name: Name of the file listener.

• Location: Project directory in which the file listener is created.

• Description: Optional. Description of the file listener

• Status: Status of the file listener: Enabled or Disabled. A disabled file listener does not listen to files on the designated folder.

• Runtime Environment: Runtime environment that contains the Secure Agent on which file listener runs.

• Source Type: Type of source to which file listener listens. Select one of the following source types:
  • Server: File listener listens to the server.
  • Connector: File listener listens to the connection.

• Connection Type: Type of the connection to which the file listener listens. The following connection types are supported.
  • Local folder
  • Advanced FTP V2
  • Advanced FTPS V2
  • Advanced SFTP V2
  • Amazon S3 V2
  • Microsoft Azure Blob Storage V3
  • Google Cloud Storage V2
  • Hadoop Files V2
  • Microsoft Azure Data Lake Storage Gen1 V3
  • Microsoft Azure Data Lake Storage Gen2

3.3 Configure File Listener rules

Configure the following parameters to define listener rules:

• Folder Path: Path to the folder to which the file listener listens.

• Pattern Type: Determines the pattern of the file name to which the file listener listens. Select one of the following patterns:
  • Wildcard
  • Regex
  • Indicator File

• File Pattern: File name pattern to which the file listener listens.
  • Wildcard: Use wildcard patterns of file name.
  • Regex: Use a regular expression to match the file name pattern.
  • Indicator File: Use the file name to which the file listener listens.

• Check for files in sub-folders: Indicates whether the file listener checks for files in sub-folders under the folder to which it listens.

• Post Action: Determines the action the file listener must perform after the file listener listens to the events. You can select the post action as Delete only if the file pattern is an Indicator file. Default is None.

• Notify when file: Determines when the file listener must send notifications to the services that are registered to it. You can select as many options as required.
  • Arrived: Sends notifications when files arrive at the folder to which the file listener listens.
  • Updated: Sends notifications when files in the folder to which the file listener listens are updated.
  • Is Deleted: Sends notifications when files in the folder to which the file listener listens are deleted.

• Stop checking if rules are met: The file listener stops listening to the folder when the listener rules are met.
  • For example, if you configure the file listener to send notifications when the files in the folder to which it listens are deleted, the listener stops listening to the folder when the first event of file deletion occurs in the folder.
  • If this option is not selected, the file listener notifies the registered application on events and continues to listen for subsequent events.

• Check File stability: The file listener verifies that the entire file is copied to the folder to which it listens before notifying the registered services. Select this option if you transfer large files, where the process of writing the files to the folder is not instant.

• Stability Check Interval: Time in seconds that a file listener waits to check for file stability. For example, if the stability time is 15 seconds, a file listener verifies the status of the file after every 15 seconds.

• Notify if files exist on the first run: When the file listener runs for the first time, it sends a notification if files exist in the folder to which it listens. The Exclude events when the file listener is not running option is disabled when you select this option.

• Exclude events when the file listener is not running: Determines if you want to exclude the file events that occur when a file listener is not running. The Notify if files exist on the first run option is disabled when you select this option.

3.4 Configure File Listener Schedule

Configure the following parameters to specify a schedule:

• You can set schedule options to run a file listener daily, weekly, or monthly.
• You can set schedule options to start a file listener on a specific date and time, and seconds, based on a specific time zone and to run file listener on a reoccurring basis.

3.5 Configure File Listener Failure Management

Configure the following parameters to define Failure Management:

• Continue to run on failure: Select this option to run file listener even if the task fails with errors.
• Send a notification on failure: Select this option to receive a notification if the task fails. Enter a list of email addresses separated by comma to which you want to send notifications if a task fails.

3. Triggering Taskflow based on File Event using Connector File Listener

You can invoke a taskflow through a connector File Listener.

To trigger taskflow based on file event, you have to define the Binding type as Event and select the Connector File Listener that you have already configured as the Event Source.

When you publish the taskflow, the taskflow subscribes to the connector file listener that is configured in it. When a file event occurs, the connector file listener invokes the taskflow.

When you configure the connector file listener to listen for new files on a folder, the connector file listener invokes the associated taskflow each time a new file arrives in the specified folder.

To unsubscribe from the connector file listener, Unpublish the taskflow.

4. Orchestrate Taskflow execution through File Events using File Watch step

You can add a File Watch Task step to a taskflow to listen to files in a defined location and monitor file events. Using File Watch step you can wait for a file to arrive at a particular location and then consume the file in a subsequent steps of taskflow.

In the File Watch Task step, you have to select an existing Connector File Listener.

The below image shows a taskflow that contains a File Watch Task step. When a file event occurs, the file watch task runs and sends the file event details such as a list of arrived, updated, and deleted files along with the file details. The taskflow then proceeds to the subsequent steps.

4.1 Configuring Connector File Listener in File Watch step

Under File Watch Task step properties, you can select an existing file listener with the connector source type that you want to add to the taskflow. When you select a file listener, you see a description and output fields for the file listener.

4.2 Configuring Timeout of File Watch Task

If a file event does not occur, by default, the taskflow waits for 5 minutes. After that, the File Watch Task step completes and the taskflow proceeds to the subsequent steps.

You can override the time out value using Time Out field under Input Fields tab of File Watch step.

4.3 Configuring Timers to a File Watch step

Use Events properties of File Watch task step to add timers to a file watch task.

When you add a timer to a File Watch Task step, a new branch appears. Add an event to this branch and specify whether you want the event to run At a specific time or After an interval.

In the following image, the event on the timer branch is a Notification step that occurs five minutes after the main file watch task starts

In the example, the timer branch runs five minutes after File Watch Task starts. If File Watch Task finishes before five minutes after it is triggered, the timer branch is not executed.

You can select the Interrupting option if you want the timer to interrupt the main file watch task.

The following image shows an interrupting timer set to occur five minutes after the main file watch task starts. If File Watch Task completes before the timer, the taskflow executes only File Watch Task and ends.

5. Behavioral differences of a File Listener

Certain behavioural differences exist when a file listener is configured as Event Source with Binding type as Event in Taskflow versus File Watch Task step of a taskflow.

6. Summary

1. A File Listener component in Informatica Cloud listens to file events on a defined location and file pattern.
2. A file event is detected when new files arrive to the monitored folder or the files in the monitored folder are updated or deleted.
3. When you want the taskflow to be triggered based on a file event, configure Binding type as Event with file listener as Event source. The taskflow subscribes to file listener and is invoked automatically whenever a file event occurs.
4. You can add a File Watch Task step to a taskflow to monitor files in a defined location.
5. You can use multiple file watch steps in a taskflow.
6. The file listener associated to a file watch step starts only when the taskflow is invoked and stops when the first occurrence of a file event or a timeout occurs.

Related Articles: