topic Re: Multiline Event being split into multiple events in Getting Data In

Multiline Event being split into multiple events

alanzchan — Wed, 30 Sep 2020 01:36:30 GMT

I have a multiline event that's being split into multiple events. I've tried LINE_BREAKER, BREAK_ONLY_BEFORE, and BREAK_ONLY_BEFORE_DATE. I've increased the value of TRUNCATE and MAX_EVENTS as well, but it still splits the log into two different events.

When I try on the UI to add data and input a test file with the same events, it indexes correctly by default.

Am I missing something here. Why does it show perfectly fine when I add data and input a test file, but when the data is coming from a UF with the same configurations in props.conf, it splits my event?

EDIT: adding sample event and props.conf file.

Here is an example of a sample event:

TR=2019-08-04 16:28:55,565|[TRACE]|[c.m.e.m.a.MainTracing]|[XNIO-2 task-xxx]| - Request Payload[payment/authorize/v4/authorize]: <?xml version="1.0" encoding="UTF-8"?><AuthorizeRequest>
<authAction>xxx</authAction>
<originInfo>
<division>xxx</division>
<storeNbr>xxx</storeNbr>
<regNbr>xxx</regNbr>
<tranNbr>xxx</tranNbr>
<seqNbr>xxx</seqNbr>
<origTransDateCCYYMMDD>xxx</origTransDateCCYYMMDD>
<origTransTimeHHMMSS>xxx</origTransTimeHHMMSS></originInfo>
<tenderType>xxx</tenderType>
<creditCard>
<acctFormat>xxx</acctFormat>
<acctNbr>xxx</acctNbr>
<panLast4>xxx</panLast4>
<acctLength>xxx</acctLength>
<aesInfo>
<aesDivision>xxx</aesDivision>
<aesStoreNbr>xxx</aesStoreNbr>
<aesKeyVersion>xxx</aesKeyVersion></aesInfo>
<entryMode>xxx</entryMode>
<propAcctType>xxx</propAcctType>
<promoCode>xxx</promoCode>
<propCardDivision>xxx</propCardDivision></creditCard>
<amount>xxx</amount>
<associateInfo>
<sellingAssociate1>xxx</sellingAssociate1>
<ringingAssociate>xxx</ringingAssociate></associateInfo>
<terminalInfo>
<pinEntryCapability>xxx</pinEntryCapability>
<terminalEntryCapability>xxx</terminalEntryCapability>
<model>xxx</model>
<applicationName>xxx</applicationName>
<applicationVersion>xxx</applicationVersion></terminalInfo>
<deptNbr>xxx</deptNbr>
<customerInfo>
<id>
<idType>xxx</idType>
<idNbrAesVersion>xxx</idNbrAesVersion>
<idNbr>xxx</idNbr>
<idState>xxx</idState>
<idEntryMode>xxx</idEntryMode>
<idValidationStatus>xxx</idValidationStatus></id></customerInfo>
<cliEnabled>xxx</cliEnabled>
<clientDataList>
<clientData id="cardholderId">xxx</clientData></clientDataList></AuthorizeRequest>

My event will always be split right after the first AuthorizeRequest in the second line.

Here is my props.conf stanza:

[my_sourcetype]
MAX_TIMESTAMP_LOOKAHEAD = 30
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)TR=
TIME_PREFIX = TR=
TIME_FORMAT = %Y-%m-%d %H:%M:%S,%N
TRUNCATE = 100000
MAX_EVENTS = 100000

Re: Multiline Event being split into multiple events

marycordova — Mon, 05 Aug 2019 21:57:24 GMT

you needs to post your props.conf config and a sample data set and a sample of the truncated events

Re: Multiline Event being split into multiple events

alanzchan — Tue, 06 Aug 2019 13:42:52 GMT

Hi marycordova,

I've edited my original post to include a sample log and props.conf.

Re: Multiline Event being split into multiple events

alanzchan — Tue, 06 Aug 2019 14:13:30 GMT

I'd like to emphasize that it works perfectly fine when using the add data feature on Splunk user interface. The logs are split only when they're are coming in from the universal forwarder (with the same configurations in props.conf).

Re: Multiline Event being split into multiple events

alanzchan — Tue, 06 Aug 2019 16:11:30 GMT

I also tried this configuration:

[my_sourcetype]
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE_DATE = true
TIME_PREFIX = TR=
TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3N
TRUNCATE = 50000
MAX_EVENTS = 200

It still does not work. Am I doing something wrong? Other multiline events are indexing fine. This one seems to be different for some reason.

Re: Multiline Event being split into multiple events

richgalloway — Tue, 06 Aug 2019 19:39:35 GMT

Is the props.conf file on the indexers or the UF? It should be the indexers. If you have an intermediate HF, put the props.conf there, too.

Re: Multiline Event being split into multiple events

alanzchan — Tue, 06 Aug 2019 19:54:17 GMT

The configuration file is on the indexers. There are no intermediate forwarders. I'm starting to think I have to do SEDCMD to remove all the new lines and carriage returns for this to work.

Re: Multiline Event being split into multiple events

woodcock — Thu, 08 Aug 2019 01:36:27 GMT

Your settings are correct so it must be something else. If you are doing a sourcetype override/overwrite, you must use the ORIGINAL value, NOT the new value. You must deploy your settings to the first full instance(s) of Splunk that handle the events (usually either the HF tier if you use one, or else your Indexer tier), restart all Splunk instances there, send in new events (old events will stay broken), then test using _index_earliest=-5m to be absolutely certain that you are only examining the newly indexed events.

Also, go onto the device that has the logs and do a tail -f <path to file and filename here> and watch as events come in. There may be a long pause at the point Splunk is breaking the event, if so, there are settings that can help, but the right thing to do is force your logging application not to pause like that.

Re: Multiline Event being split into multiple events

MuS — Thu, 08 Aug 2019 04:04:43 GMT

check for typos of the sourcetype name in the stanza in props.conf, because it is actually case sensitive in props.conf.

cheers, MuS

Re: Multiline Event being split into multiple events

alanzchan — Thu, 08 Aug 2019 13:46:37 GMT

The long pause might be the problem. What are ways to go about resolving this issue?

Re: Multiline Event being split into multiple events

woodcock — Wed, 14 Aug 2019 17:09:50 GMT

See here,
https://docs.splunk.com/Documentation/Splunk/latest/Admin/Inputsconf
specifically:

time_before_close = <integer>
* The amount of time, in seconds, that the file monitor must wait for
  modifications before closing a file after reaching an End-of-File
  (EOF) marker.
* Tells the input not to close files that have been updated in the
  past 'time_before_close' seconds.
* Default: 3.

multiline_event_extra_waittime = <boolean>
* By default, the file monitor sends an event delimiter when:
  * It reaches EOF of a file it monitors and
  * Ihe last character it reads is a newline.
* In some cases, it takes time for all lines of a multiple-line event to
  arrive.
* Set to "true" to delay sending an event delimiter until the time that the
  file monitor closes the file, as defined by the 'time_before_close' setting,
  to allow all event lines to arrive.
* Default: false.

Re: Multiline Event being split into multiple events

alanzchan — Wed, 14 Aug 2019 17:34:14 GMT

Thanks woodcock,

I was speaking with Splunk Support today and made these changes. Your comment was really helpful.