Getting Data In

Split multi-line events

Explorer

I have a file that has multiple multi line events. Each event is broken up into "INFO: ---" or "ERROR: ---"

ERROR: ---
blahNewsLetter: N
birthdate: 1947-10-25
countryId: 1
createdOn: 2011-05-31 13:40:46
...
INFO: ---
blahNewsLetter: ~
birthdate: 0000-00-00
countryId: ~
createdOn: 2011-05-31 13:40:48
...

My props.conf

[dblog]
SHOULD_LINEMERGE = true
# force splunk to detec multiline events
BREAK_ONLY_BEFORE = (.*)(INFO|ERROR):

My inputs.conf

[monitor:///var/log/dblog.log]
disabled = false
index = blah
sourcetype = dblog
blacklist = (\.(gz|bz2|z|zip)$)
followTail = 1

This does not work. It is splitting the file up into events that have lines similar to the "createdOn: 2011-05-31 13:40:48" lines.

modifiedOn: 2011-05-31 13:40:48
postalCode: 1111
promoCode: ~
requestDetail: |-
  <request>
  <billToEmail>null</billToEmail>
  <billToFirstName>Name</billToFirstName>
  <billToLastName>Name</billToLastName>
  <billToStreet1>null</billToStreet1>
  <billToStreet2>null</billToStreet2>
...

and

createdOn: 2011-05-31 13:40:48
email: myemail@email.com

Any help would be appreciated. And I have also tried using BREAK_ONLY_BEFORE_DATE = false with same result. This is currently on the forwarder.

1 Solution

Builder

lcasey001,

I would recommend using LINE_BREAKER.

[dblog]
SHOULD_LINEMERGE = false
# force splunk to detec multiline events
LINE_BREAKER = ([\r\n]+)(?:INFO|ERROR):

Update: These configurations need to be on your indexer. I would also recommend modifying Splunk's date/time properties since there is not a timestamp in first 150 characters of your event.

[dblog]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)(?:INFO|ERROR):
TIME_PREFIX = createdOn:\s+
TIME_FORMAT = %Y-%m-%d %H:%M:%S

View solution in original post

Builder

lcasey001,

I would recommend using LINE_BREAKER.

[dblog]
SHOULD_LINEMERGE = false
# force splunk to detec multiline events
LINE_BREAKER = ([\r\n]+)(?:INFO|ERROR):

Update: These configurations need to be on your indexer. I would also recommend modifying Splunk's date/time properties since there is not a timestamp in first 150 characters of your event.

[dblog]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)(?:INFO|ERROR):
TIME_PREFIX = createdOn:\s+
TIME_FORMAT = %Y-%m-%d %H:%M:%S

View solution in original post

Builder

Not sure if there is a specific list, but any props related to line breaking, time parsing, or of type TRANSFORMS- are performed @ the indexing/full forwarder layer of your Splunk infrastructure.

0 Karma

Explorer

Where can one obtain that list for future reference? I did not see anything in the props.conf.spec file.

0 Karma

Explorer

Thanks !!! That was it. It needed to be done on the indexer and not forwarder. So is there a list of things that needs to be done on the indexer vs forwarder regarding props.conf ?

0 Karma

Builder

Line breaking and time parsing are done @ the indexer or Full forwarder, not Universal Forwarder/Light Forwarder. Please move these configurations to your indexer(s).

0 Karma

Explorer

These changes have been on the Universal Forwarder where the log file is hosted. I have other settings on the forwarder for other log files, mostly setting sourcetypes in the props.conf file. Should this all be done on the indexer?

0 Karma

Explorer

Made your recommended change, restarted the forwarder, and got the same result.

5/31/11
3:31:17.000 PM  

    modifiedOn: 2011-05-31 15:31:17
    postalCode: 11111
    promoCode: ''
    requestDetail: |-
      <request>
      <email>email@email.net</email>
      <userName>username</userName>
      <firstName>firstname</firstName>
      <lastName>lastname</lastName>
      <language>en</language>
Show all 18 lines
0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!