Cummins File Transfer Processing - BART

Background

A process has been developed to transfer files between Cummins and external sites. This process is bi-directional so that files may be transferred to or from Cummins. This process is also round-trip so that the sending site will be notified of any issues with any files transferred. This documents the basics of this transfer process and how to set up for transfers. Hence the acronym for this application is BART - Bi-directional And Round-trip Transfers.

Since every site may send files and every site may receive files, the setup is basically symmetric. However one area of diffence is that some sites may have special processing for certain types of files they receive. For example sites sending PAM data to Cummins want the files processed into the PAM system and responses from the loading process returned. In this case the Cummins setup will include information on what processing is to be done for this type of file.

This process is based on the Cummins system which permits files to be transferred through the Cummins firewall. This system is known as CFG - Cummins File Gateway. While transfers through this server are based on sftp, this gateway system is highly configured and arbitrary transfer schemas are not supported. For example adding a new site requires a moderate amount of CFG setup.

Configuration

There are 2 required configuration files and 1 optional configuration file to setup a site for transfers.

Transfer Directory Setups

A standard directory structure must be used when transferring files. One structure is used for files to be sent and another for files being received. These structures may be located under the same base directory for convenience but is not a requirement.

The directory structure for sending files parallels the structure used for tranMan and the external data manager, for those familar with those processes, except for being more adaptable.

send_directory
   send_file_type_1
      complete
         test_cell_1
         test_cell_2
         ...
      hold
         test_cell_1
         test_cell_2
         ...
      ready
         test_cell_1
         test_cell_2
         ...
   send_file_type_2
      complete
         ...
      hold
         ...
      ready
         ...
   send_file_type_3
      ...
The "send_directory" is specified in the site configuration file. The code then locates the various file type directories. To add a new file type, create the directory with the appropriate structure above and restart the transfer program.

A file to be sent to the remote site should be placed in the appropriate "ready" directory based on the file type and the test cell the file is coming from. When the file has been transferred it will be moved from the "ready" directory to the corresponding location under the "hold" directory. Once a reply is received from the remote site that the file was successfully transferred and if processing was involved successfully processed, the file will be moved from "hold" to the appropriate location under the "complete" directory. If the file failed to transfer or if processing was unsuccessful, the file will remain in the "hold" directory and an error message file with the same base name as the file will be created.

To receive files of various types create the following structure.

receive_directory
   problems
   transfer
"receive_directory" will be specified in the site configuration file. Any transfer related problems will generate files in the "problems" directory. "transfer" is a working directory used by the code but otherwise not of interest to users. Other directories under receive_directory are created automatically as files for storage are received. In this case the directory name will be the received file type.

Transfer File Naming Convention

Files being transferred have a naming convention. CFG does not permit dynamic directory creation. So rather than use sub-directories like the transfer program has for sending and receiving, this structure information is encoded into the file name being sent and received through CFG.

site#testcell#type#name
The sending process encodes this name based on the directory structure where the file was located. The receiving process decodes this name and proceeds to process the file. The received file might only be for storage and the file gets stored in a directory structure corresponding to the parsed values. Received files might be designated for processing rather than storage and the processing is performed.

Note: This convention implies that none of the names involved in this process may contain a sharp (#) character since this is used as a delimiter.

Installation

This application is distributed as a zip file. Unzipping it will create a directory named "cummins_transfers". In this directory are a number of files.

Installation steps.

Operations

Once installed this system is fairly simple to operate. This process should run continuously. Once it starts the program will check CFG for any files to be received. If it finds any it will process them and send reply messages to the sender. It then checks configured directories for any files to be sent to the remote site. If it doesn't find any received files or any files to be sent it will sleep for a period and then wake up and check again.

Since this program checks a global resource for files to process it may run on any server. However it should only have one executing instance at any time. If there are two instances running at the same time, they would end up fighting each other over files in CFG to process.

There is a "start_cummins_transfers" script which will start this program into execution. To gracefully stop this process the command "stop_cummins_transfers" may be used.

Periodically sites should check files under the "problems" directories to see if there are transfer or data problems. Periodically sites should clean up files under the "complete" directories. On a more regular basis files under the "hold" directory should be checked. These files either indicate communication problems or problems with the data being sent. In either case, these should be check on a regular basis since these files indicate issues which probably need to be addressed.

Sample Configuration Files

Generally this document will be used by people at a remote site needing to send files to Cummins. The following sample files use YOUR_SITE_NAME as the local site id and "Cummins" as the remote site id. When installing this system at a remote site, anything with a "YOUR" prefix should be replaced by site specific values. These examples specify files and directories using Linux / UNIX style. If being setup for Windows, standard Windows style should be used. But note that backslash characters (\) should be doubled in this case. For example:

   /home/myhome/transfers  Linux/UNIX
   versus
   c:\\myhome\\transfers   Windows

transfer_config.txt

The distributed package of files which include this documentation also contains a transfer_config.txt file. See it for further details.

Cummins_config.txt

The distributed package of files which include this documentation also contains a Cummins_config.txt file. See it for further details.

data_types.xml

If this configuration file is not present then no received files will receive special processing. Files will be transferred to and from Cummins and replies processed but that's all. For non-Cummins sites this is usually appropriate. However Cummins will have various file type processing specified. This information is not further detailed here since it is expected that only programmers for this process need to be aware of the details of this information. Also there is presently no use of this file at remote sites.