CSV

Receiver Sender

This module implements CSV service and poller.

docker pull registry.goingrid.io/services/csv:v0.1.0

This service implements a csv file service.

Table of Content

Properties

Like every other service, the csv service includes all properties of the service configuration and since it listens on input the input configuration and also the output config.

In addition those are the following file service properties for the service:

Data

Parameter Default value Description
“*” Fields Values per Titles or Fields Configuration

Controls

The following Controls are available for this module

Control Description
“filename” “Sets the Output filename when writeing to a CSV File”
“titles” Contains the Titles from the Input Document read from the first line.
“filesum” Contains the Filesum of the currently imported file

API

The csv file service api lets you do the following:

  • Poll changes from Directory and Scan for Files. Read the CSV File and Parse it. Delete or Move it afterwards in to another Directory.
  • Append Data to an existing CSV File or create a new one.

Poll changes from file / dir

With this config

    -ServerURL "natspw@natsserver"
    -OutputChannel "debug_csv"
    -WorkDir "/tmp/transfer"
    -BackupDir "/tmp/backup"
    -InputPattern ".*\\.csv"
    -Delimiter ";"
    -FirstLineTitles

The files matched by the Input Pattern will be parsed and generate the following events:

If CommandAttribute is set, the Start Event will look like this:

Class: csv
Operation: input
Control:
   filename:
       testdata
    filesum:
       344632edb6af63b91fce9a579a6a5854
    cmd:
       start  

Class: csv
Operation: input
Control:
   filename:
       testdata
   filesum:
      344632edb6af63b91fce9a579a6a5854
   titles:
       field1
       field2
       field3
Data:
    field1:
       value1
    field2:
       value2
    field3
       value3
       value4

Class: csv
Operation: input
Control:
    filename:
       testdata
    filesum:
       344632edb6af63b91fce9a579a6a5854
    cmd:
       end

Append to file

The input interface data object can contains following:

    -ServerURL "natspw@natsserver"
    -InputChannel "csv_write"
    -WorkDir "/tmp/transfer"
    -Delimiter ";"
    -FirstLineTitles
    -Fields "feld1,feld2,feld3"

Class: csv
Operation: write
Control:
    filename:
      myFile.csv
Data:
    feld1:
       wert1
    feld2:
       wert2
    feld3:
       wert3
       wert4

This will create a CSV File like this:

feld1;feld2;feld3
wert1;wert2;wert3,wert4

Service configuration

Like every other service, the log service includes all properties of the service configuration.

The following parameters are specific to this service. They can be either used as environment variables (all uppercase) or within the CLI (- as prefix).

Read File Parameters

This Service needs and Output Channel.

Parameter Default Description
“WorkDir” "/transfer" “Path where the Inport or Export Files should be located”
“Delimiter” ";" “The CSV Field Delimiter to be used”
“ValueDelimiter” "," “Delimiter for Multivalue Datefields, to use to join the Output for the CSV”
“Fields” "" “Field to write to or read from the CSV. Comma delimited List of Fields to write/read”
“FirstLineTitles” true “Use Title from first line of CSV instead of Fields for Reading. If Fields are Empty use all, if Fields is provided, cherry Pick”
“CommandAttribute” "cmd" “Attribute name where we put commands into. The Service will give start / run / stop in the command. For Pre or Post Jobs to be run on imports”
“FirstLineTitles” false “Use Title from first line of CSV instead of Fields for Reading. If Fields are Empty use all, if Fields is provided, cherry Pick”
“CommandAttribute” "cmd" “Attribute name where we put commands into. The Worker will give start / run / stop in the command. For Pre or Post Jobs to be run on imports”
“DoFileSum” true “Add a ‘filesum’ Attribute in the Document with the current File Filesum”
“Comment” "#" “Lines starting with this Rune are ignored”
“FieldPerRecord” -1 “Fields ro read per line. -1=ignore / 0=all same length / >0 all must have this length”
“Lazy” false “Try to load bad Quoted Files”
“Trim” false “Trim Spaces in Fields”
“DefaultClass” "csv" “Set a default class for the ingrid request”

Write File Parameters

This service needs and Input Channel.

Parameter Default Description
“WorkDir” "/transfer" “Path where the Inport or Export Files should be located”
“Delimiter” ";" “The CSV Field Delimiter to be used”
“ValueDelimiter” "," “Delimiter for Multivalue Datefields, to use to join the Output for the CSV”
“Fields” "" “Field to write to or read from the CSV. Comma delimited List of Fields to write/read”
“FirstLineTitles” false “Write the Fields as Titles onto the first line”
“BackupDir” "/backup" “Path where the Imported Files should be moved to. If Import is done. Original File will be deleted”
“InputPattern” ".*\\.csv$" “Pattern of Files to Process (Regex), Hint: When using in a Docker Compose File, you need to use $$ first $ is escape char”
“PollInterval” 10 “Interval in Seconds to Scan the Directory”
“UseCRLF” false “Use Windows newline instead of \n”