SFTP File Transfer Task

Background

There are times where files in the standard external data manager file structure, see below, need to be transferred to a different system. This documents a task to perform that operation.

This task is based on the standard external data manager (ext.dat.man) file structure. The standard file structure is:

<data_type>/ready/<test_cell_1>
                 /<test_cell_2>
                 /...
<data_type>/hold/<test_cell_1>
                /<test_cell_2>
                /...
<data_type>/complete/<test_cell_1>
                    /<test_cell_2>
                    /...
Each test cell directory under "ready" will contain input files for that test cell. Test cells are responsible for getting files to the appropriate location under the "ready" directory. After being sent, a file will be moved from "ready" to the appropriate test cell directory under "complete". Files do not go into "hold" like some other ext.dat.man applications. A file either gets transferred successfully and moves into the "complete" directory or it fails to transfer and remains in the "ready" directory. Files under the "complete" directory should be cleaned up from time to time.

Managing Load Processing

start_sftp_transfers <property_file> - This script will start this task into operation for a specific property file specified configuration. The property file specifies the source ext.dat.man file structure containing files to be sent and the remote system and location where the files are to be sent. A documented sample configuration file should be included in the package containing this writeup. See it for details.

However one configuration option requires additional discussion. cyflex.extdatman.remoteIsExtDatMan indicates if the remote system directory structure is an ext.dat.man file structure. This operates in coordination with cyflex.extdatman.targetDirectory. There are two cases: