Getting Data In

Multiline Event being split into multiple events

alanzchan
Path Finder

I have a multiline event that's being split into multiple events. I've tried LINE_BREAKER, BREAK_ONLY_BEFORE, and BREAK_ONLY_BEFORE_DATE. I've increased the value of TRUNCATE and MAX_EVENTS as well, but it still splits the log into two different events.

When I try on the UI to add data and input a test file with the same events, it indexes correctly by default.

Am I missing something here. Why does it show perfectly fine when I add data and input a test file, but when the data is coming from a UF with the same configurations in props.conf, it splits my event?

EDIT: adding sample event and props.conf file.

Here is an example of a sample event:

TR=2019-08-04 16:28:55,565|[TRACE]|[c.m.e.m.a.MainTracing]|[XNIO-2 task-xxx]| - Request Payload[payment/authorize/v4/authorize]: <?xml version="1.0" encoding="UTF-8"?><AuthorizeRequest>
<authAction>xxx</authAction>
<originInfo>
<division>xxx</division>
<storeNbr>xxx</storeNbr>
<regNbr>xxx</regNbr>
<tranNbr>xxx</tranNbr>
<seqNbr>xxx</seqNbr>
<origTransDateCCYYMMDD>xxx</origTransDateCCYYMMDD>
<origTransTimeHHMMSS>xxx</origTransTimeHHMMSS></originInfo>
<tenderType>xxx</tenderType>
<creditCard>
<acctFormat>xxx</acctFormat>
<acctNbr>xxx</acctNbr>
<panLast4>xxx</panLast4>
<acctLength>xxx</acctLength>
<aesInfo>
<aesDivision>xxx</aesDivision>
<aesStoreNbr>xxx</aesStoreNbr>
<aesKeyVersion>xxx</aesKeyVersion></aesInfo>
<entryMode>xxx</entryMode>
<propAcctType>xxx</propAcctType>
<promoCode>xxx</promoCode>
<propCardDivision>xxx</propCardDivision></creditCard>
<amount>xxx</amount>
<associateInfo>
<sellingAssociate1>xxx</sellingAssociate1>
<ringingAssociate>xxx</ringingAssociate></associateInfo>
<terminalInfo>
<pinEntryCapability>xxx</pinEntryCapability>
<terminalEntryCapability>xxx</terminalEntryCapability>
<model>xxx</model>
<applicationName>xxx</applicationName>
<applicationVersion>xxx</applicationVersion></terminalInfo>
<deptNbr>xxx</deptNbr>
<customerInfo>
<id>
<idType>xxx</idType>
<idNbrAesVersion>xxx</idNbrAesVersion>
<idNbr>xxx</idNbr>
<idState>xxx</idState>
<idEntryMode>xxx</idEntryMode>
<idValidationStatus>xxx</idValidationStatus></id></customerInfo>
<cliEnabled>xxx</cliEnabled>
<clientDataList>
<clientData id="cardholderId">xxx</clientData></clientDataList></AuthorizeRequest>

My event will always be split right after the first AuthorizeRequest in the second line.

Here is my props.conf stanza:

[my_sourcetype]
MAX_TIMESTAMP_LOOKAHEAD = 30
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)TR=
TIME_PREFIX = TR=
TIME_FORMAT = %Y-%m-%d %H:%M:%S,%N
TRUNCATE = 100000
MAX_EVENTS = 100000
0 Karma
1 Solution

woodcock
Esteemed Legend

Your settings are correct so it must be something else. If you are doing a sourcetype override/overwrite, you must use the ORIGINAL value, NOT the new value. You must deploy your settings to the first full instance(s) of Splunk that handle the events (usually either the HF tier if you use one, or else your Indexer tier), restart all Splunk instances there, send in new events (old events will stay broken), then test using _index_earliest=-5m to be absolutely certain that you are only examining the newly indexed events.

Also, go onto the device that has the logs and do a tail -f <path to file and filename here> and watch as events come in. There may be a long pause at the point Splunk is breaking the event, if so, there are settings that can help, but the right thing to do is force your logging application not to pause like that.

View solution in original post

MuS
SplunkTrust
SplunkTrust

check for typos of the sourcetype name in the stanza in props.conf, because it is actually case sensitive in props.conf.

cheers, MuS

woodcock
Esteemed Legend

Your settings are correct so it must be something else. If you are doing a sourcetype override/overwrite, you must use the ORIGINAL value, NOT the new value. You must deploy your settings to the first full instance(s) of Splunk that handle the events (usually either the HF tier if you use one, or else your Indexer tier), restart all Splunk instances there, send in new events (old events will stay broken), then test using _index_earliest=-5m to be absolutely certain that you are only examining the newly indexed events.

Also, go onto the device that has the logs and do a tail -f <path to file and filename here> and watch as events come in. There may be a long pause at the point Splunk is breaking the event, if so, there are settings that can help, but the right thing to do is force your logging application not to pause like that.

alanzchan
Path Finder

The long pause might be the problem. What are ways to go about resolving this issue?

0 Karma

woodcock
Esteemed Legend

See here,
https://docs.splunk.com/Documentation/Splunk/latest/Admin/Inputsconf
specifically:

time_before_close = <integer>
* The amount of time, in seconds, that the file monitor must wait for
  modifications before closing a file after reaching an End-of-File
  (EOF) marker.
* Tells the input not to close files that have been updated in the
  past 'time_before_close' seconds.
* Default: 3.

multiline_event_extra_waittime = <boolean>
* By default, the file monitor sends an event delimiter when:
  * It reaches EOF of a file it monitors and
  * Ihe last character it reads is a newline.
* In some cases, it takes time for all lines of a multiple-line event to
  arrive.
* Set to "true" to delay sending an event delimiter until the time that the
  file monitor closes the file, as defined by the 'time_before_close' setting,
  to allow all event lines to arrive.
* Default: false.

alanzchan
Path Finder

Thanks woodcock,

I was speaking with Splunk Support today and made these changes. Your comment was really helpful.

0 Karma

alanzchan
Path Finder

I'd like to emphasize that it works perfectly fine when using the add data feature on Splunk user interface. The logs are split only when they're are coming in from the universal forwarder (with the same configurations in props.conf).

0 Karma

alanzchan
Path Finder

I also tried this configuration:

[my_sourcetype]
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE_DATE = true
TIME_PREFIX = TR=
TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3N
TRUNCATE = 50000
MAX_EVENTS = 200

It still does not work. Am I doing something wrong? Other multiline events are indexing fine. This one seems to be different for some reason.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Is the props.conf file on the indexers or the UF? It should be the indexers. If you have an intermediate HF, put the props.conf there, too.

---
If this reply helps you, Karma would be appreciated.
0 Karma

alanzchan
Path Finder

The configuration file is on the indexers. There are no intermediate forwarders. I'm starting to think I have to do SEDCMD to remove all the new lines and carriage returns for this to work.

0 Karma

marycordova
SplunkTrust
SplunkTrust

you needs to post your props.conf config and a sample data set and a sample of the truncated events

@marycordova

alanzchan
Path Finder

Hi marycordova,

I've edited my original post to include a sample log and props.conf.

0 Karma
Get Updates on the Splunk Community!

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...