topic Re: Split multi-line events in Getting Data In

Split multi-line events

lcasey001 — Mon, 28 Sep 2020 09:37:59 GMT

I have a file that has multiple multi line events. Each event is broken up into "INFO: ---" or "ERROR: ---"

ERROR: ---
blahNewsLetter: N
birthdate: 1947-10-25
countryId: 1
createdOn: 2011-05-31 13:40:46
...
INFO: ---
blahNewsLetter: ~
birthdate: 0000-00-00
countryId: ~
createdOn: 2011-05-31 13:40:48
...

My props.conf

[dblog]
SHOULD_LINEMERGE = true
# force splunk to detec multiline events
BREAK_ONLY_BEFORE = (.*)(INFO|ERROR):

My inputs.conf

[monitor:///var/log/dblog.log]
disabled = false
index = blah
sourcetype = dblog
blacklist = (\.(gz|bz2|z|zip)$)
followTail = 1

This does not work. It is splitting the file up into events that have lines similar to the "createdOn: 2011-05-31 13:40:48" lines.

modifiedOn: 2011-05-31 13:40:48
postalCode: 1111
promoCode: ~
requestDetail: |-
  <request>
  <billToEmail>null</billToEmail>
  <billToFirstName>Name</billToFirstName>
  <billToLastName>Name</billToLastName>
  <billToStreet1>null</billToStreet1>
  <billToStreet2>null</billToStreet2>
...

and

createdOn: 2011-05-31 13:40:48
email: myemail@email.com

Any help would be appreciated. And I have also tried using BREAK_ONLY_BEFORE_DATE = false with same result. This is currently on the forwarder.

Re: Split multi-line events

hazekamp — Tue, 31 May 2011 19:24:19 GMT

lcasey001,

I would recommend using LINE_BREAKER.

[dblog]
SHOULD_LINEMERGE = false
# force splunk to detec multiline events
LINE_BREAKER = ([\r\n]+)(?:INFO|ERROR):

Update: These configurations need to be on your indexer. I would also recommend modifying Splunk's date/time properties since there is not a timestamp in first 150 characters of your event.

[dblog]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)(?:INFO|ERROR):
TIME_PREFIX = createdOn:\s+
TIME_FORMAT = %Y-%m-%d %H:%M:%S

Re: Split multi-line events

lcasey001 — Tue, 31 May 2011 19:35:08 GMT

Made your recommended change, restarted the forwarder, and got the same result.

5/31/11
3:31:17.000 PM  

    modifiedOn: 2011-05-31 15:31:17
    postalCode: 11111
    promoCode: ''
    requestDetail: |-
      <request>
      <email>email@email.net</email>
      <userName>username</userName>
      <firstName>firstname</firstName>
      <lastName>lastname</lastName>
      <language>en</language>
Show all 18 lines

Re: Split multi-line events

lcasey001 — Tue, 31 May 2011 20:50:17 GMT

These changes have been on the Universal Forwarder where the log file is hosted. I have other settings on the forwarder for other log files, mostly setting sourcetypes in the props.conf file. Should this all be done on the indexer?

Re: Split multi-line events

hazekamp — Tue, 31 May 2011 20:53:06 GMT

Line breaking and time parsing are done @ the indexer or Full forwarder, not Universal Forwarder/Light Forwarder. Please move these configurations to your indexer(s).

Re: Split multi-line events

lcasey001 — Tue, 31 May 2011 21:11:00 GMT

Thanks !!! That was it. It needed to be done on the indexer and not forwarder. So is there a list of things that needs to be done on the indexer vs forwarder regarding props.conf ?

Re: Split multi-line events

lcasey001 — Wed, 01 Jun 2011 00:52:30 GMT

Where can one obtain that list for future reference? I did not see anything in the props.conf.spec file.

Re: Split multi-line events

hazekamp — Wed, 01 Jun 2011 03:38:05 GMT

Not sure if there is a specific list, but any props related to line breaking, time parsing, or of type TRANSFORMS- are performed @ the indexing/full forwarder layer of your Splunk infrastructure.