topic How to remove an invalid line breaker from syslog before indexing? in Getting Data In

How to remove an invalid line breaker from syslog before indexing?

vlours — Thu, 29 Sep 2016 03:42:39 GMT

Hi everyone,

I've got an application sending data to splunk, which are split over multiple lines instead to keep everything on the same line.

When I redirect my data to a file instead of splunk, I can find that the ascii code #012 is sent as part of the string.

Example:

... #012Change details       : #012filewrite#012 ...

Which are split in multiples lines in splunk:

...
9/29/16 3:25:30.000 AM  filewrite
host = xxx.xxx.xxx.xxx source = udp:3514 sourcetype = syslog
9/29/16 3:25:30.000 AM  Change details       :
host = xxx.xxx.xxx.xxx source = udp:3514 sourcetype = syslog
...

Is there any way to replace the ASCII code #012 before to index it into splunk ?

I've try to add this config in my props.conf, but it did not solved it.

[syslog]
LINE_BREAKER=#012
SHOULD_LINEMERGE=true

And also this one:

[syslog]
SEDCMD-fim = s/\#012/ /g

Thanks for your support.

Re: How to remove an invalid line breaker from syslog before indexing?

somesoni2 — Thu, 29 Sep 2016 05:10:26 GMT

The #012 is ascii code for \n, so that is why events are getting split (default LINE_BREAKER is ([\r\n]+)). You second configuration with SEDCMD will not work as SEDCMD executes after events are broken.

How does your data looks like when you redirect it to a file (sample entries, mask any sensitive data)? You probably have to setup correct LINE_BREAKER (assuming you're data is directly coming to indexer/heavy forwarder) to split it correctly.

Re: How to remove an invalid line breaker from syslog before indexing?

vlours — Tue, 29 Sep 2020 11:11:30 GMT

Hi somesoni2,

Actually, I did not test it with the default LINE_BREAKER, as I was fighting with my #012 value.
Setting the default value LINE_BREAKER and the SHOULD_LINEMERGE is working fine.

So, I've put the following in my props.conf:
[syslog]
LINE_BREAKER=([\r\n]+)
SHOULD_LINEMERGE=true

Thanks a lot.
Have a great day.

Re: How to remove an invalid line breaker from syslog before indexing?

vlours — Tue, 29 Sep 2020 11:11:33 GMT

As explain by @somesoni2, the #012 is is ascii code for \n

So to solve it, I've include the following in my props.conf:
[syslog]
LINE_BREAKER=([\r\n]+)
SHOULD_LINEMERGE=true

My data are now merged correctly in one line.