File

Receiver Sender

This module implements a _template worker and poller.

docker pull registry.goingrid.io/services/file:v0.0.1

Table of Content

Service configuration
Service capabilities
- Poll changes from file / dir
- Append to file
Template functions

Service configuration

Properties

Like every other service, the log service includes all properties of the service configuration and since it listens on input the input configuration and also the output config.

In addition those are the following file service properties for the service:

Parameter	Default value	Description
“WorkDir”	`"/transfer"`	“Path where the Inport or Export Files should be located”
“Template”	`""`	“Template for File Import or Export”
“BackupDir”	`"/backup"`	“Path where the Imported Files should be moved to. If Import is done orig File will be deleted”
“CommandAttribute”	`"cmd"`	“Attribute name where we put commands into. The Worker will give start / run / stop in the command. For Pre or Post Jobs to be run on imports”
“InputPattern”	`".*\\.dat"`	“Pattern of Files to Process (Regex”
“PollInterval”	`10`	“Interval in Seconds to Scan the Directory”
“TemplateRequestTimeout”	`10`	“Timeout for Fetching Template from Hive”
“DoFileSum”	`true`	“Add a ‘filesum’ Attribute in the Document with the current File Filesum”
“StartTag”	`"[["`	“Template Start Tag”
“StopTag”	`"]]"`	“Template Stop Tag”

Controls

The following Controls are available for this module

Control	Description
“filename”	“Sets the Output filename when writeing to a File”
“cmd”	“Contains the Command start or stop (CommandAttribute)”
“filesum”	“Contains the Current FileSum”

Service capabilities

The file service api lets you do the following:

Poll changes from Directory and Scan for Files. Read the File and Parse it. Delete or Move it afterwards in to another Directory
Append Data or Create a File.

Poll changes from file / dir

Config

Service will poll to the Working Directory.

The files matched by the Input Pattern will be parsed and prcessed by the template:

-ServerURL "natspw@natsserver"
-OutputChannel "debug_file"
-WorkDir "/tmp/transfer"
-BackupDir "/tmp/backup"
-InputPattern ".*\\.dat"

Template

Class: file
Operation: input
Control:
    filename:
       testdata
    filesum:
       344632edb6af63b91fce9a579a6a5854
Data:
    lines:
       asdfsadf
       sdfsdaf

You can pass a Template to the worker with -Template

Examples:

-Template "hive://myFileTemplate.tmpl"
-Template "/templates/myFileTemplate.tmpl"
-Template "[[ $data := newData ]][[ $data = addValues $data \"lines\" . ]][[ sendEvent $data ]]"

In the Template all data from file is passed as array of strings. Each line will be a string.

To make one event per line you can use a Template like this:

[[ range . ]]
  [[ data := newData ]]
  [[ $data = addValues $data "line" . ]]
  [[ sendEvent $data ]]
[[ end ]]

You could also parse the line further more to make multiple string fields etc.

Start Event

If CommandAttribute is set, the Start Event will look like this:

Class: file
Operation: input
Control:
    filename:
       testdata
    filesum:
       344632edb6af63b91fce9a579a6a5854
    cmd:
       start

End Event

Class: file
Operation: input
Control:
    filename:
       testdata
    filesum:
       344632edb6af63b91fce9a579a6a5854
    cmd:
       end

Kubernetes manifest

apiVersion: apps/v1
kind: Deployment
metadata:
  name: file-poll-service
spec:
  selector:
    matchLabels:
      app: file-poll-service
  template:
    metadata:
      labels:
        app: file-poll-service
    spec:
      containers:
      - name: file-poll-service
        env:
        - name: name
          value: file-poll-service
        - name: SERVERHOST
          value: core-ingrid-hive # CHANGE IF REQUIRED
        - name: LOGLEVEL
          value: debug
        - name: LOGFORMAT
          value: TEXT/DEFAULT
        - name: LOGSENSITIVE
          value: "false"
        - name: WORKDIR
          value: /data/transfer
        - name: BACKUPDIR
          value: /data/backup
        - name: INPUTPATTERN
          value: fileOne.txt
        - name: POLLINTERVALL
          value: "10"
        - name: DOFILESUM
          value: "true"
        - name: OUTPUTCHANNEL
          value: file_write-filetwo
        - name: TEMPLATE
          value: |-
            [[ range . ]]
            [[ $data := newData ]]
            [[ $data  = setValue $data "text" . ]]
            [[ $data  = setValue $data "file" "two" ]]
            [[ sendEvent $data ]]
            [[ end ]]
        - name: SERVERAUTH
          valueFrom:
            secretKeyRef:
              name: core-ingrid-serverkey # CHANGE IF REQUIRED
              key: serverkey # CHANGE IF REQUIRED
        image: registry-dev.goingrid.io/services/file:latest
        imagePullPolicy: Always
        resources:
          limits:
            cpu: 200m
            memory: 32Mi
          requests:
            cpu: 50m
            memory: 16Mi
        volumeMounts:
        - mountPath: /data
          name: data
      serviceAccountName: core-ingrid # CHANGE IF REQUIRED
      serviceAccount: core-ingrid # CHANGE IF REUQIRED
      restartPolicy: Always
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: file-claim

Append to file

All fields will be passed to the Template. You can create whatever Filecontent from the Data you like.

The Output of the Template will be appended to the File.

Config

    -ServerURL "natspw@natsserver"
    -InputChannel "debug_file"
    -WorkDir "/tmp/transfer"

The files matched by the Input Pattern will be parsed and generate the following events:

Kubernetes manifest

apiVersion: apps/v1
kind: Deployment
metadata:
  name: file-append-fileone-service
spec:
  selector:
    matchLabels:
      app: file-append-fileone-service
  template:
    metadata:
      labels:
        app: file-append-fileone-service
    spec:
      containers:
      - name: file-append-fileone-service
        env:
        - name: name
          value: file-append-fileone-service
        - name: SERVERHOST
          value: core-ingrid-hive # CHANGE IF REQUIRED
        - name: LOGLEVEL
          value: debug
        - name: LOGFORMAT
          value: TEXT/DEFAULT
        - name: LOGSENSITIVE
          value: "false"
        - name: WORKDIR
          value: /data
        - name: INPUTCHANNEL
          value: file_append-fileone
        - name: TEMPLATE
          value: |-
            [[ (index .Data.index 0 ) ]];[[ (index .Data.text 0 )]] [[ print "\n" ]]

        - name: SERVERAUTH
          valueFrom:
            secretKeyRef:
              name: core-ingrid-serverkey # CHANGE IF REQUIRED
              key: serverkey # CHANGE IF REQUIRED
        image: registry-dev.goingrid.io/services/file:latest
        imagePullPolicy: Always
        resources:
          limits:
            cpu: 200m
            memory: 32Mi
          requests:
            cpu: 50m
            memory: 16Mi
        volumeMounts:
        - mountPath: /data
          name: data
      serviceAccountName: core-ingrid # CHANGE IF REQUIRED
      serviceAccount: core-ingrid # CHANGE IF REUQIRED
      restartPolicy: Always
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: file-claim

Template

The Default Template for Output is

[[- range .Data ]]
  [[- .]];
[[ end ]]

[[ .Data.Field1 ]];[[.Data.Field2 ]]
Some Constant Text
[[ .Data.Field1 ]];[[.Data.Field2 ]]
Some More Stuff

Template functions

The following Template Functions are implemented. sendEvent is the IMPORTANT one.

Sprig has been added to. You can use all the Sprig Template functions

Function Name	Parameters	Result
“sendEvent”	data Data	bool
“after”	value string, a string	string
“listContains”	arr []string, search string	bool
“debug”	objs …interface{}	bool
“timeParse”	layout, value string	Time, error
“int”	a interface{}	int
“mapData”	inmap map[string]string, data pkgmsg.Data	pkgmsg.Data
“mapStrictData”	inmap map[string]string, data pkgmsg.Data	pkgmsg.Data
“prefixReplace”	inmap map[string]string, data pkgmsg.Data	pkgmsg.Data
“suffixReplace”	inmap map[string]string, data pkgmsg.Data	pkgmsg.Data
“marshal”	v interface{}	string
“marshalIndent”	v interface{}	string
“marshalJson”	v interface{}	string
“marshalJsonIndent”	v interface{}	string
“marshalYaml”	data interface{}	string
“minus”	a, b int	int
“newMap”		map[string]string
“nilToBlank”	err error	string
“noescape”	str string	template.HTML
“plus”	a, b int	int
“replaceLine”	input, search, newline string	string
“replaceLineAndIdent”	input, search, newline string	string
“setMapEntry”	inmap map[string]string, key string, value string	map[string]string
“string”	v interface{}	string
“toLower”	s string	string
“toUpper”	s string	string
“unmarshal”	source string	interface{}
“unmarshalYaml”	source string	interface{}
“newData”:		Data
“addValue”:	data Data, name string, value string	Data
“addValues”:	data Data, name string, values []string	Data
“clearValue”:	data Data, name string	Data
“getValue”:	data Data, name string	string
“getValues”:	data Data, name string	[]string
“setValue”:	data Data, name string, value string	Data
“setValues”:	data Data, name string, values []string	Data
“getKeys”:	data Data	[]string
“set”:	key string, value interface{}	string
“get”:	key string	interface{}
“stripSuffix”	value string, suffix string	string
“stripPrefix”	value string, prefix string	string