Multiline Event being split into multiple events

[alanzchan]

I have a multiline event that's being split into multiple events. I've tried LINE_BREAKER, BREAK_ONLY_BEFORE, and BREAK_ONLY_BEFORE_DATE. I've increased the value of TRUNCATE and MAX_EVENTS as well, but it still splits the log into two different events.

When I try on the UI to add data and input a test file with the same events, it indexes correctly by default.

Am I missing something here. Why does it show perfectly fine when I add data and input a test file, but when the data is coming from a UF with the same configurations in props.conf, it splits my event?

EDIT: adding sample event and props.conf file.

Here is an example of a sample event:

TR=2019-08-04 16:28:55,565|[TRACE]|[c.m.e.m.a.MainTracing]|[XNIO-2 task-xxx]| - Request Payload[payment/authorize/v4/authorize]: <?xml version="1.0" encoding="UTF-8"?><AuthorizeRequest>
<authAction>xxx</authAction>
<originInfo>
<division>xxx</division>
<storeNbr>xxx</storeNbr>
<regNbr>xxx</regNbr>
<tranNbr>xxx</tranNbr>
<seqNbr>xxx</seqNbr>
<origTransDateCCYYMMDD>xxx</origTransDateCCYYMMDD>
<origTransTimeHHMMSS>xxx</origTransTimeHHMMSS></originInfo>
<tenderType>xxx</tenderType>
<creditCard>
<acctFormat>xxx</acctFormat>
<acctNbr>xxx</acctNbr>
<panLast4>xxx</panLast4>
<acctLength>xxx</acctLength>
<aesInfo>
<aesDivision>xxx</aesDivision>
<aesStoreNbr>xxx</aesStoreNbr>
<aesKeyVersion>xxx</aesKeyVersion></aesInfo>
<entryMode>xxx</entryMode>
<propAcctType>xxx</propAcctType>
<promoCode>xxx</promoCode>
<propCardDivision>xxx</propCardDivision></creditCard>
<amount>xxx</amount>
<associateInfo>
<sellingAssociate1>xxx</sellingAssociate1>
<ringingAssociate>xxx</ringingAssociate></associateInfo>
<terminalInfo>
<pinEntryCapability>xxx</pinEntryCapability>
<terminalEntryCapability>xxx</terminalEntryCapability>
<model>xxx</model>
<applicationName>xxx</applicationName>
<applicationVersion>xxx</applicationVersion></terminalInfo>
<deptNbr>xxx</deptNbr>
<customerInfo>
<id>
<idType>xxx</idType>
<idNbrAesVersion>xxx</idNbrAesVersion>
<idNbr>xxx</idNbr>
<idState>xxx</idState>
<idEntryMode>xxx</idEntryMode>
<idValidationStatus>xxx</idValidationStatus></id></customerInfo>
<cliEnabled>xxx</cliEnabled>
<clientDataList>
<clientData id="cardholderId">xxx</clientData></clientDataList></AuthorizeRequest>

My event will always be split right after the first AuthorizeRequest in the second line.

Here is my props.conf stanza:

[my_sourcetype]
MAX_TIMESTAMP_LOOKAHEAD = 30
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)TR=
TIME_PREFIX = TR=
TIME_FORMAT = %Y-%m-%d %H:%M:%S,%N
TRUNCATE = 100000
MAX_EVENTS = 100000

[MuS]

check for typos of the sourcetype name in the stanza in props.conf, because it is actually case sensitive in props.conf.

cheers, MuS

[woodcock · Accepted solution]

Your settings are correct so it must be something else. If you are doing a sourcetype override/overwrite, you must use the ORIGINAL value, NOT the new value. You must deploy your settings to the first full instance(s) of Splunk that handle the events (usually either the HF tier if you use one, or else your Indexer tier), restart all Splunk instances there, send in new events (old events will stay broken), then test using _index_earliest=-5m to be absolutely certain that you are only examining the newly indexed events.

Also, go onto the device that has the logs and do a tail -f <path to file and filename here> and watch as events come in. There may be a long pause at the point Splunk is breaking the event, if so, there are settings that can help, but the right thing to do is force your logging application not to pause like that.

[alanzchan]

The long pause might be the problem. What are ways to go about resolving this issue?

[woodcock]

See here,
https://docs.splunk.com/Documentation/Splunk/latest/Admin/Inputsconf
specifically:

time_before_close = <integer>
* The amount of time, in seconds, that the file monitor must wait for
  modifications before closing a file after reaching an End-of-File
  (EOF) marker.
* Tells the input not to close files that have been updated in the
  past 'time_before_close' seconds.
* Default: 3.

multiline_event_extra_waittime = <boolean>
* By default, the file monitor sends an event delimiter when:
  * It reaches EOF of a file it monitors and
  * Ihe last character it reads is a newline.
* In some cases, it takes time for all lines of a multiple-line event to
  arrive.
* Set to "true" to delay sending an event delimiter until the time that the
  file monitor closes the file, as defined by the 'time_before_close' setting,
  to allow all event lines to arrive.
* Default: false.

[alanzchan]

Thanks woodcock,

I was speaking with Splunk Support today and made these changes. Your comment was really helpful.

[alanzchan]

I'd like to emphasize that it works perfectly fine when using the add data feature on Splunk user interface. The logs are split only when they're are coming in from the universal forwarder (with the same configurations in props.conf).

[alanzchan]

I also tried this configuration:

[my_sourcetype]
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE_DATE = true
TIME_PREFIX = TR=
TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3N
TRUNCATE = 50000
MAX_EVENTS = 200

It still does not work. Am I doing something wrong? Other multiline events are indexing fine. This one seems to be different for some reason.

[richgalloway]

Is the props.conf file on the indexers or the UF? It should be the indexers. If you have an intermediate HF, put the props.conf there, too.

[alanzchan]

The configuration file is on the indexers. There are no intermediate forwarders. I'm starting to think I have to do SEDCMD to remove all the new lines and carriage returns for this to work.

[marycordova]

you needs to post your props.conf config and a sample data set and a sample of the truncated events

[alanzchan]

Hi marycordova,

I've edited my original post to include a sample log and props.conf.