Indirect File Loading in Informatica Cloud (IICS)

Introduction

The ability to process multiple source files of same structure and properties through single source transformation in a mapping is called Indirect file loading.

If you are from Powercenter background you might be already familiar with Indirect file loading concept. Let’s discuss how that can be implemented in Informatica Cloud.

Direct File Loading

Normally when we want to process a flat file as a source in Informatica Cloud, you select a Flat File connection in the source and select the flat file you wanted to process as a source. Here the Source Type by default will be selected as ‘Single object’. This is referred as a Direct loading technique.

For example if you wanted to process employee data from flat file into a target, you select the details as shown below in source in Direct loading technique.

Indirect File Loading

Let’s say you are receiving multiple employee information files. These files are with same structure and flat file properties. Then there is no need to create a separate mapping for each source file. Informatica provides an option to load multiple flat files of same file structure and properties through a single mapping and this functionality is referred as Indirect loading.

Instead of passing the actual source files as source, we pass a file which holds the information of all source filenames as source. Hence this load type is referred as Indirect loading.

Guidelines for creating file list:

• All files in the file list must share same file structure and properties.
• If no file directory path is mentioned along with file name, the mapping assumes the files are present in the directory path mentioned in the Source connection.
• Else if file path is mentioned, files are picked up from the mentioned directory.
• Every path must be local to the secure agent.

There are 3 different ways in which we can implement Indirect file loading in Informatica Cloud.

1. Using the File List as the Source Type in Source transformation.
2. Using the Command as the Source Type in Source transformation.
3. Using the Data Integration Quick Start Bundle.

Let’s discuss each type in detail

1. Indirect File Loading using File List

In the source transformation, under Source Type select File List.

Under the Object, select the file which contains the list of source files to be processed.

List file contents

Source file contents

Once target is configured, trigger the mapping and all the records from all the source files will be loaded into the target.

2. Indirect File Loading using Command Source Type

In the source transformation, under Source Type select Command.

In this method you need to provide a Sample Object as shown below. Select any one of the source file as a sample object or create a custom sample object.

Under Command provide the script name along with the directory path.

Windows Script contents:

Linux Script contents:

If you are running on Linux sever, the Command will be like /Informatica/Scripts/Get_Filelist.sh and script contents as below.

cd /Informatica/Scripts/
ls Employees_*.csv

Once target is configured, trigger the mapping and all the records from all the source files will be loaded into the target.

3. Indirect File Loading using the Data Integration Quick Start Bundle

3.1 Installing Data Integration Quick Start Bundle

Navigate to Informatica Administrator, in the menu on the left side select Add-On Bundles.

Click on Available Bundles, select the Data integration Quick start bundle and Install it.

Now the bundle must be available under Installed Bundles.

3.2 Creating Mapping Configuration Task using installed Bundle

Now under Projects, a new project Add-On Bundles is created.

Go to Add-On Bundles> Data Integration Quick Start

Right Click on Multiple File Processing and select New Mapping Task.

No mapping required. The bundle will create only a Mapping Task and handles the functionality through Mapping task.

3.3 Configuring the Mapping task to perform Indirect Loading

In the Definition section select the Task Name and Runtime Environment

Under the Sources, select Source Connection and Source Object (sample file)

Under the Targets, select Target Connection and Target object.

Under the Other Parameters section, provide the file name which holds the list of all source files.

Do not mention any source directory path of the file. The path should be same as the path mentioned in the Source connection.

Finally map the target fields and save the mapping task.

Trigger the mapping task and all the source rows must be now loaded into target.

Conclusion

Personally I do not prefer the final method of implementation. It requires installing an Add-On Bundle and lengthy process compared to other methods.

Anyhow here are all the various methods by which we can implement Indirect file loading in Informatica Cloud (IICS).