Output Files to S3¶

Overview¶

Digital Process Factory can output files to an S3 bucket, similar to the S3 output for PLOSSYS Output Engine. This is mainly relevant for SaaS installations, but not limited to them.

The s3.copyFromRli node working unit copies the files listed in an RLI to a configured S3 bucket.

Prerequisites¶

An existing S3 bucket that matches your infrastructure. Digital Process Factory does not create it.
Access credentials for the bucket: an access key ID and a secret access key.
The s3.copyFromRli node working unit. It ships as part of seal-dpf-wustandard and is included by default.
A workflow that implements the s3.copyFromRli node working unit.

Configure the S3 Connection¶

Specify the S3 connect with the following environment keys for the seal-dpf-wustandard service.
- S3_BUCKET: S3 bucket for uploading files. Mandatory.
- S3_ACCESS_KEY_ID: Access key ID for uploading files to S3. Mandatory.
- S3_SECRET_ACCESS_KEY: Secret access key for uploading files to S3. Mandatory.
- S3_REGION: S3 region for uploading files. Optional. Default: eu-central-1.
- S3_ENDPOINT: URL of the S3 endpoint. Optional. Only needed for an S3-compatible service that is not accessible via AWS's default per-region endpoints, for example MinIO.
- S3_FILE_NAME_TEMPLATE: Template for target file names on S3, with placeholders for RLI item parameters. Optional. Default: original file names.
- S3_BASE_DIR_TEMPLATE: Template for the job directory name on S3, with placeholders for RLI parameters. Optional. Default: original job directory.
- S3_METADATA_KEYS: RLI item metadata keys, separated with commas, for filtering S3 object metadata when it exceeds S3's 2 KB metadata size limit. Optional.
Hint - subdirectories

The job directory name (S3_BASE_DIR_TEMPLATE or baseDirTemplate) and the target file names (S3_FILE_NAME_TEMPLATE or the TargetFileName RLI item parameter) may contain /, which is interpreted as subdirectory on S3.

Hint - corresponding WU parameters

The input parameters region, bucket, accessKeyId, secretAccessKey, endpoint, fileNameTemplate, baseDirTemplate, metadataKeys overwrite the corresponding environment variable listed above.
1. If you don't already have a workflow that calls s3.copyFromRli, create one for testing, for example test.s3CopyFromRli:
```
<?xml version="1.0" encoding="ISO-8859-1"?>

<DPF>
   <config>
      <workflows>
         <test.s3CopyFromRli COMMENT="" NAME="Test S3 CopyFromRli" DPFINT-Interactive="0" PROJECT="test" VERSION="$Id$" INTERNALNAME="test.s3CopyFromRli" DPFINT-Customer="0" DPFINT-Type="WorkflowDescription">
            <MAIN BEGIN="CopyToS3" NAME=""/>
            <CopyToS3 ERROR="END" NEXT="END" NAME="s3.copyFromRli" TYPE="WU" POSY="400" POSX="300" COMMENT="Upload the files listed in the RLI to S3.">
               <DEFAULTS/>
               <MAPPING/>
               <CONNECTIONS ERROR="DIRECT" NEXT="DIRECT"/>
            </CopyToS3>
            <END POSY="100" POSX="200" COMMENT="" NAME="END" TYPE="STOP"/>
         </test.s3CopyFromRli>
      </workflows>
   </config>
</DPF>
```
  Hint - workflow deployment
  
  Save the workflow file in the customer-specific directory, refer to Customer-Specific Files on Linux or Windows. Then restart seal-dpf-process-manager, refer to Administrate the DPF Services. If you deploy the workflow to MongoDB instead, no restart is needed, refer to Administrate DPF in SaaS.
2. Call the workflow with DPF Batch Client to validate your configuration.
  Example - calling a workflow using DPF Batch Client in a PowerShell
```
dpfclient -wf test.s3CopyFromRli -protocol http -host localhost:4303 -f "test.rli"
```
  To overwrite set environment variables for the job, use the -s option:
```
dpfclient -wf test.s3CopyFromRli -protocol http -host localhost:4303 -s bucket="blubb" -f "test.rli"
```

Filter RLI Lines¶

s3.copyFromRli can upload only the files of RLI lines that match a filter. The filter parameters have the same names as for the classic RLI filter, but the filter behaves differently in several cases, refer to Differences to DPF 3.x.

Set the following parameters for the working unit, for example in the MAPPING element of its workflow position:

RlicFilter: name of the RLI column to check. Case and underscores in column names do not matter, neither in RlicFilter nor in the RLI.
FilterOperator: comparison operator. Optional. Default: eq.
- eq, ==: the column value equals one of the filter values. Both operators compare text.
- ne, !=: the column value differs from all filter values.
- =~: the column value matches one of the regular expressions.
- !~: the column value matches none of the regular expressions.
FilterValue: one or more values to compare the column value with. Separate several values with spaces. Mandatory. Without FilterValue, the working unit ignores this filter and all following filters.

Regular expressions use JavaScript syntax and always match the whole column value, as if enclosed in ^…$. To match a part of the value, add .*, for example NOMEN.*. An invalid regular expression ends the job with an error.

Example - upload only lines with IS_DIST_FILE set to Y

<MAPPING RlicFilter="IS_DIST_FILE" FilterValue="Y"/>

Combine Several Filters¶

Add further filters with numbered parameters: RlicFilter2, FilterOperator2, FilterValue2, then RlicFilter3 and so on. The first filter has no number. By default, a line must match all filters.

AtLeastOneFilterMustMatch: specifies whether a line must match only one of the filters. Optional.
- Y, J, or 1, case-insensitive: one matching filter is sufficient.
- Any other value: all filters must match. Default.

Differences to DPF 3.x¶

The classic RLI filter is used in DPF 3.x and by classic working units such as Rli.copyFiles. The filter of s3.copyFromRli differs from it:

Topic	Classic RLI filter	`s3.copyFromRli`
Several values in `FilterValue`	Separated with commas.	Separated with spaces.
Regular expressions	Match a part of the value.	Match the whole value.
`/…/i` syntax, for example `/\.pdf$/i`	Supported.	Not supported, never matches.
Perl syntax in regular expressions, for example `(?i)`	Supported.	Invalid. The job ends with an error.
Operators	All Perl comparison operators and file test operators.	Only `eq`, `==`, `ne`, `!=`, `=~`, `!~`.
`==`, `!=`	Compare numbers, for example `01` equals `1`.	Compare text, for example `01` differs from `1`.
Empty `FilterValue`	Compares with an empty value.	The filter and all following filters are ignored.
`ne`, `!~` with several values	Match if the column value differs from at least one value.	Match if the column value differs from all values.
`ne`, `!~` on a missing column	No line matches. The job log shows a warning.	All lines match. No warning.
`FilterPrefix`	Supported.	Not supported.
`AtLeastOneFilterMustMatch`	`y`, `j`, `x`, or `1` anywhere in the value.	Exactly `y`, `j`, or `1`.

Output Parameters¶

CurrRliFileName: new RLI file.
IGNORED_ERRORS: number of ignored errors if IGNORE_MISSING_FILES is set to Y.