topic Re: forwarder creating multiline event from big file transferred via rsycn on slow channel in Getting Data In

forwarder creating multiline event from big file transferred via rsycn on slow channel

gots — Sun, 14 May 2017 16:26:59 GMT

We have big file with events in each line of file. Every minute file transfers via rsync to forwarder with new events.
Events format:

timestamp="XXXX" login="YYY" action="ZZZ"

Size of file about 10Mb, with 15000 lines.
Sometimes (1-2 times per minute) forwarder compile one event from multiple (usually 2) lines.
Configs from forwarder:
inputs.conf:

[monitor:///file.log]
disabled = false
followTail = 1
index = f_logs
sourcetype = f_logs
multiline_event_extra_waittime = true
time_before_close = 10

props.conf:

[f_logs]
TIME_PREFIX = timestamp=\"
TIME_FORMAT = %s
SHOULD_LINEMERGE = true
MAX_EVENTS = 1

What i have to do to solve this problem?

Re: forwarder creating multiline event from big file transferred via rsycn on slow channel

anand_singh17 — Fri, 09 Jun 2017 12:10:08 GMT

Two ways to solve:

option 1-
Let the forwarder create multi-line. Use "transaction" keyword to group similar based on common field you prepare for.

Option2-
try using trail method for the log. Let rsync write to file.log, but use another crontab to send data to another file - rysnclistner.log.
Now listen to this log file.

Reason:
1. Changes the frequency of reading data for fowarder
2. You can use it in form of Queue and data purges as new entries keeps come. (FIFO)

Hope any of the option can help. But most important, we also would like to see some sample _raw to be able to decide.

Thank You