Example Config File
Below are two example config YAML files. The first tails the minifi-app.log, send the tailed log and provenance data back to a secure instance of NiFi. The second uses a series of processors to tail the app log, routes off only lines that contain "WriteAheadFlowFileRepository" and puts it as a file in the "" directory.
MiNiFi Config Version: 1
Flow Controller:
name: MiNiFi Flow
comment:
Core Properties:
flow controller graceful shutdown period: 10 sec
flow service write delay interval: 500 ms
administrative yield duration: 30 sec
bored yield duration: 10 millis
FlowFile Repository:
partitions: 256
checkpoint interval: 2 mins
always sync: false
Swap:
threshold: 20000
in period: 5 sec
in threads: 1
out period: 5 sec
out threads: 4
Provenance Repository:
provenance rollover time: 1 min
Content Repository:
content claim max appendable size: 10 MB
content claim max flow files: 100
always sync: false
Component Status Repository:
buffer size: 1440
snapshot frequency: 1 min
Security Properties:
keystore: /tmp/ssl/localhost-ks.jks
keystore type: JKS
keystore password: localtest
key password: localtest
truststore: /tmp/ssl/localhost-ts.jks
truststore type: JKS
truststore password: localtest
ssl protocol: TLS
Sensitive Props:
key:
algorithm: PBEWITHMD5AND256BITAES-CBC-OPENSSL
provider: BC
Processors:
- name: TailFile
class: org.apache.nifi.processors.standard.TailFile
max concurrent tasks: 1
scheduling strategy: TIMER_DRIVEN
scheduling period: 1 sec
penalization period: 30 sec
yield period: 1 sec
run duration nanos: 0
auto-terminated relationships list:
Properties:
File to Tail: logs/minifi-app.log
Rolling Filename Pattern: minifi-app*
Initial Start Position: Beginning of File
Connections:
- name: TailToS2S
source name: TailFile
source relationship name: success
destination name: 8644cbcc-a45c-40e0-964d-5e536e2ada61
max work queue size: 0
max work queue data size: 1 MB
flowfile expiration: 60 sec
queue prioritizer class: org.apache.nifi.prioritizer.NewestFlowFileFirstPrioritizer
Remote Processing Groups:
- name: NiFi Flow
comment:
url: https://localhost:8090/nifi
timeout: 30 secs
yield period: 10 sec
Input Ports:
- id: 8644cbcc-a45c-40e0-964d-5e536e2ada61
name: tailed log
comments:
max concurrent tasks: 1
use compression: false
Provenance Reporting:
comment:
scheduling strategy: TIMER_DRIVEN
scheduling period: 30 sec
destination url: https://localhost:8090/
port name: provenance
originating url: http://${hostname(true)}:8081/nifi
use compression: true
timeout: 30 secs
batch size: 1000Flow Controller:
name: MiNiFi Flow
comment:
Core Properties:
flow controller graceful shutdown period: 10 sec
flow service write delay interval: 500 ms
administrative yield duration: 30 sec
bored yield duration: 10 millis
max concurrent threads: 1
FlowFile Repository:
partitions: 256
checkpoint interval: 2 mins
always sync: false
Swap:
threshold: 20000
in period: 5 sec
in threads: 1
out period: 5 sec
out threads: 4
Content Repository:
content claim max appendable size: 10 MB
content claim max flow files: 100
always sync: false
Component Status Repository:
buffer size: 1440
snapshot frequency: 1 min
Security Properties:
keystore: /tmp/ssl/localhost-ks.jks
keystore type: JKS
keystore password: localtest
key password: localtest
truststore: /tmp/ssl/localhost-ts.jks
truststore type: JKS
truststore password: localtest
ssl protocol: TLS
Sensitive Props:
key:
algorithm: PBEWITHMD5AND256BITAES-CBC-OPENSSL
provider: BC
Processors:
- name: TailAppLog
class: org.apache.nifi.processors.standard.TailFile
max concurrent tasks: 1
scheduling strategy: TIMER_DRIVEN
scheduling period: 10 sec
penalization period: 30 sec
yield period: 1 sec
run duration nanos: 0
auto-terminated relationships list:
Properties:
File to Tail: logs/minifi-app.log
Rolling Filename Pattern: minifi-app*
Initial Start Position: Beginning of File
- name: SplitIntoSingleLines
class: org.apache.nifi.processors.standard.SplitText
max concurrent tasks: 1
scheduling strategy: TIMER_DRIVEN
scheduling period: 0 sec
penalization period: 30 sec
yield period: 1 sec
run duration nanos: 0
auto-terminated relationships list:
- failure
- original
Properties:
Line Split Count: 1
Header Line Count: 0
Remove Trailing Newlines: true
- name: RouteErrors
class: org.apache.nifi.processors.standard.RouteText
max concurrent tasks: 1
scheduling strategy: TIMER_DRIVEN
scheduling period: 0 sec
penalization period: 30 sec
yield period: 1 sec
run duration nanos: 0
auto-terminated relationships list:
- unmatched
- original
Properties:
Routing Strategy: Route to 'matched' if line matches all conditions
Matching Strategy: Contains
Character Set: UTF-8
Ignore Leading/Trailing Whitespace: true
Ignore Case: true
Grouping Regular Expression:
WALFFR: WriteAheadFlowFileRepository
- name: PutFile
class: org.apache.nifi.processors.standard.PutFile
max concurrent tasks: 1
scheduling strategy: TIMER_DRIVEN
scheduling period: 0 sec
penalization period: 30 sec
yield period: 1 sec
run duration nanos: 0
auto-terminated relationships list:
- failure
- success
Properties:
Directory: ./
Conflict Resolution Strategy: replace
Create Missing Directories: true
Maximum File Count:
Last Modified Time:
Permissions:
Owner:
Group:
Connections:
- name: TailToSplit
source name: TailAppLog
source relationship name: success
destination name: SplitIntoSingleLines
max work queue size: 0
max work queue data size: 1 MB
flowfile expiration: 60 sec
queue prioritizer class: org.apache.nifi.prioritizer.NewestFlowFileFirstPrioritizer
- name: SplitToRoute
source name: SplitIntoSingleLines
source relationship name: splits
destination name: RouteErrors
max work queue size: 0
max work queue data size: 1 MB
flowfile expiration: 60 sec
queue prioritizer class: org.apache.nifi.prioritizer.NewestFlowFileFirstPrioritizer
- name: RouteToS2S
source name: RouteErrors
source relationship name: matched
destination name: PutFile
max work queue size: 0
max work queue data size: 1 MB
flowfile expiration: 60 sec
queue prioritizer class: org.apache.nifi.prioritizer.NewestFlowFileFirstPrioritizer
Provenance Reporting:
comment:
scheduling strategy: TIMER_DRIVEN
scheduling period: 30 sec
destination url: https://localhost:8080/
port name: provenance
originating url: http://${hostname(true)}:8081/nifi
use compression: true
timeout: 30 secs
batch size: 1000
