Getting Data In

Line breaking doesn'twork and my event is divided in 2 events

laraspatavcogni
Engager

the log is parsed in bad way.
that's the props.conf:
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)Data:\s\d{14}
MAX_EVENTS = 256
TRUNCATE = 10000
TIME_PREFIX = ^Data:\s
TIME_FORMAT = %d%m%Y%H%M%S
MAX_TIMESTAMP_LOOKAHEAD = 25

that's the log:

Data: 29052020113001
Numero file in /data/In/ = 0
Numero file in /data/In/IMS/ = 0
Numero file in /data/archive/ = 7
Processi:
a is running
b is running
platform is running
-Controllo sftp
-pippo:
Connected to .
sftp> bye
-pluto:
Connected to .
sftp> bye
-casa:
Connected to .
sftp> bye
-SMC:
Connected to .
sftp> bye
-Datalake:
Connected to .
sftp> bye
-Controllo System Log ultime 48h

no rows selected

i need to have all this log in one splunk event. with that configuration, splunk parse the log in two events:
alt text

Labels (1)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

BREAK_ONLY_BEFORE and LINE_BREAKER don't go together. Use LINE_BREAKER when SHOULD_LINEMERGE is false; otherwise, use BREAK_ONLY_BEFORE. Try these props.

SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)Data\:\s
MAX_EVENTS = 256
TRUNCATE = 10000
TIME_PREFIX = ^Data\:\s
TIME_FORMAT = %d%m%Y%H%M%S
MAX_TIMESTAMP_LOOKAHEAD = 14
---
If this reply helps you, Karma would be appreciated.

View solution in original post

0 Karma

to4kawa
Ultra Champion
[ your_sourcetype ]
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)Data:
NO_BINARY_CHECK=true
TIME_FORMAT=%d%m%Y%H%M%S
TIME_PREFIX=Data:\s
MAX_TIMESTAMP_LOOKAHEAD=14

props.conf
@richgalloway 's setting is right.
why don't you reboot splunk?

0 Karma

laraspatavcogni
Engager

i have rebooted splunk but doesn't work.. i don't know why.. i've never had this problems in props configuration. This log is produced by a shell script.. i need to modify it in some way for delete any "line break" or null value or something else for taking the entire log in an event?

0 Karma

richgalloway
SplunkTrust
SplunkTrust

BREAK_ONLY_BEFORE and LINE_BREAKER don't go together. Use LINE_BREAKER when SHOULD_LINEMERGE is false; otherwise, use BREAK_ONLY_BEFORE. Try these props.

SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)Data\:\s
MAX_EVENTS = 256
TRUNCATE = 10000
TIME_PREFIX = ^Data\:\s
TIME_FORMAT = %d%m%Y%H%M%S
MAX_TIMESTAMP_LOOKAHEAD = 14
---
If this reply helps you, Karma would be appreciated.
0 Karma

laraspatavcogni
Engager

Hi Rich, thank you per the answer but your solution doesn't work, the event is always divided into 2 parts.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Where is the break? Please show an example.

---
If this reply helps you, Karma would be appreciated.
0 Karma

laraspatavcogni
Engager

the event remain divided in the same point also changing the line breaker configuration advised by you

0 Karma

richgalloway
SplunkTrust
SplunkTrust

I asked a follow-up question a while ago, but it seems to have disappeared.

Is the last line of each event consistent? If so, we can use LINE_BREAKER = Controllo System Log \w+ \d+h([\r\n]+)Data:

---
If this reply helps you, Karma would be appreciated.
0 Karma

laraspatavcogni
Engager

i have use it but doesn't work.. the is parsed always in 2 events in the same point. at ever modify i restart the splunk service.
This log is produced by a shell script.. i need to modify it in some way for delete any "line break" or null value or something else for taking the entire log in an event?

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Is there delay between writing "Processi" and writing "A is running"? If so, how long is that delay?

---
If this reply helps you, Karma would be appreciated.
0 Karma

laraspatavcogni
Engager

the delay is present in every section delimited by label ("PIPPO:" "PLUTO:" "CASA:") the delay is around 0.2 sec. if is this the problem, how can I solve this problem?

0 Karma

richgalloway
SplunkTrust
SplunkTrust

It may not be the problem, but it's possible. Try adding this to your inputs.conf file.

multiline_event_extra_waittime = true
---
If this reply helps you, Karma would be appreciated.

laraspatavcogni
Engager

it works!!!!! thank you ❤️

0 Karma

laraspatavcogni
Engager

the log in divided in 2 events in this mode:
Data: 29052020160601
Numero file in /data/In/ = 0
Numero file in /data/In/IMS/ = 0
Numero file in /data/archive/ negli ultimi 15 min = 6
Processi
----------------DIVISION----------------
A is running
B is running
platform is running
-Controllo sftp
-PIPPO:
Connected to.
sftp> bye
-PLUTO:
Connected to
sftp> bye
-CASA:
Connected to
sftp> bye
-SMC:
Connected to
sftp> bye
-Datalake:
Connected to
sftp> bye
-Controllo System Log ultime 48h
no rows selected

0 Karma

493669
Super Champion

can you try removing below from props.conf because it seems due to below event is getting divided-

BREAK_ONLY_BEFORE = ^Data\:\s\d{14}
LINE_BREAKER = ([\r\n]+)Data\:\s\d{14}
MAX_EVENTS = 256
0 Karma

laraspatavcogni
Engager

if i removethe line breaker how splunk can know where needs to break the event?

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...