Getting Data In

Split multi-line events

lcasey001
Explorer

I have a file that has multiple multi line events. Each event is broken up into "INFO: ---" or "ERROR: ---"

ERROR: ---
blahNewsLetter: N
birthdate: 1947-10-25
countryId: 1
createdOn: 2011-05-31 13:40:46
...
INFO: ---
blahNewsLetter: ~
birthdate: 0000-00-00
countryId: ~
createdOn: 2011-05-31 13:40:48
...

My props.conf

[dblog]
SHOULD_LINEMERGE = true
# force splunk to detec multiline events
BREAK_ONLY_BEFORE = (.*)(INFO|ERROR):

My inputs.conf

[monitor:///var/log/dblog.log]
disabled = false
index = blah
sourcetype = dblog
blacklist = (\.(gz|bz2|z|zip)$)
followTail = 1

This does not work. It is splitting the file up into events that have lines similar to the "createdOn: 2011-05-31 13:40:48" lines.

modifiedOn: 2011-05-31 13:40:48
postalCode: 1111
promoCode: ~
requestDetail: |-
  <request>
  <billToEmail>null</billToEmail>
  <billToFirstName>Name</billToFirstName>
  <billToLastName>Name</billToLastName>
  <billToStreet1>null</billToStreet1>
  <billToStreet2>null</billToStreet2>
...

and

createdOn: 2011-05-31 13:40:48
email: myemail@email.com

Any help would be appreciated. And I have also tried using BREAK_ONLY_BEFORE_DATE = false with same result. This is currently on the forwarder.

1 Solution

hazekamp
Builder

lcasey001,

I would recommend using LINE_BREAKER.

[dblog]
SHOULD_LINEMERGE = false
# force splunk to detec multiline events
LINE_BREAKER = ([\r\n]+)(?:INFO|ERROR):

Update: These configurations need to be on your indexer. I would also recommend modifying Splunk's date/time properties since there is not a timestamp in first 150 characters of your event.

[dblog]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)(?:INFO|ERROR):
TIME_PREFIX = createdOn:\s+
TIME_FORMAT = %Y-%m-%d %H:%M:%S

View solution in original post

hazekamp
Builder

lcasey001,

I would recommend using LINE_BREAKER.

[dblog]
SHOULD_LINEMERGE = false
# force splunk to detec multiline events
LINE_BREAKER = ([\r\n]+)(?:INFO|ERROR):

Update: These configurations need to be on your indexer. I would also recommend modifying Splunk's date/time properties since there is not a timestamp in first 150 characters of your event.

[dblog]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)(?:INFO|ERROR):
TIME_PREFIX = createdOn:\s+
TIME_FORMAT = %Y-%m-%d %H:%M:%S

hazekamp
Builder

Not sure if there is a specific list, but any props related to line breaking, time parsing, or of type TRANSFORMS- are performed @ the indexing/full forwarder layer of your Splunk infrastructure.

0 Karma

lcasey001
Explorer

Where can one obtain that list for future reference? I did not see anything in the props.conf.spec file.

0 Karma

lcasey001
Explorer

Thanks !!! That was it. It needed to be done on the indexer and not forwarder. So is there a list of things that needs to be done on the indexer vs forwarder regarding props.conf ?

0 Karma

hazekamp
Builder

Line breaking and time parsing are done @ the indexer or Full forwarder, not Universal Forwarder/Light Forwarder. Please move these configurations to your indexer(s).

0 Karma

lcasey001
Explorer

These changes have been on the Universal Forwarder where the log file is hosted. I have other settings on the forwarder for other log files, mostly setting sourcetypes in the props.conf file. Should this all be done on the indexer?

0 Karma

lcasey001
Explorer

Made your recommended change, restarted the forwarder, and got the same result.

5/31/11
3:31:17.000 PM  

    modifiedOn: 2011-05-31 15:31:17
    postalCode: 11111
    promoCode: ''
    requestDetail: |-
      <request>
      <email>email@email.net</email>
      <userName>username</userName>
      <firstName>firstname</firstName>
      <lastName>lastname</lastName>
      <language>en</language>
Show all 18 lines
0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...