Emissions Data Loading Task

Background

Sites creating raw emissions data files need to send these files to the Cummins UNIX network to be processed. After processing, the UNIX network will send reply files back to the site submitting the files with the processing results. The results file either indicates successful processing or indicates errors and includes error messages.

CyFlex sites are already setup for sending the raw emission files and receiving the reply files. However non-CyFlex sites also need a process for doing this. This documents a process these sites may use.

Installation

Non-CyFlex sites needing to run this file transfer process will receive a file, emissions_data.zip, containing the application. Unzipping this will create a directory called "emissions_data". The complete path to and including this directory will be known as the BASE for this application. The zip file should be unpacked into the location where the application should reside. Or the upacked directory should be moved to the preferred location. In either case the final location will become the BASE location.

There are a number of updates which need to be made to the files in question before the process may be run. NOTE for Windows configurations: Windows pathes with slash characters (\) need to use 2 slash characters when updating the properties.txt file. For example F:\XYZ\ABC should be entered as F:\\XYZ\\ABC. The program reading this information will consider a single slash character as an escape character. Two slashes will then be interpreted as a single slash.

Data Structure

This process is based on the standard external data manager (extdatman) file structure. The file structure is:

<emissions_data>/ready/<test_cell_1>
                      /<test_cell_2>
                      /...
<emissions_data>/hold/<test_cell_1>
                     /<test_cell_2>
                     /...
<emissions_data>/complete/<test_cell_1>
                         /<test_cell_2>
                         /...
Each test cell directory under the "ready" directory will contain raw emission data files to be sent to the Cummins UNIX network for processing. Test cells are responsible for getting files to the appropriate test cell directory under the "ready" directory.

When a file shows up under the "ready" directory it will be sent to the UNIX network for processing. After a successful transfer, the file will be moved to the appropriate test cell directory under the "hold" directory.

Eventually a reply file will be sent by the UNIX network to a configurable location on the system which sent the data file. If the reply file indicates successful processing of the data file, the data file under the "hold" directory will be moved to the appropriate directory under the "complete" directory. If the reply file indicates unsuccessful processing, it will contain error messages identifying the issue(s). The entire reply file will be moved under the "hold" directory.

Operations

There are 2 scripts involved in managing this task.

Periodically files under the "complete" directory should be cleaned out.

Files under the "hold" directory should be monitored for error files. This is especially important if e-mails are not being sent in the event of processing errors.