topic Re: Multiple lines with the same time in Splunk Search

Multiple lines with the same time

dobarnes — Mon, 28 Sep 2020 11:26:28 GMT

I have logs from a custom application being streamed into splunk usinig a unverisal forwarder. The probelem I have there is multiple lines with the same time. See below.

19:12:51.790,16526719,TCP,2,3404,2226
19:12:51.790,66870655,TCP,10,53743,355114
19:12:51.790,199246079,TCP,5,2937,5715
19:12:51.790,281972991,TCP,2,55722,43156
19:12:51.790,282382591,TCP,11,2458,11480

I have extracted the fields of there data using the props.conf and the transforms.com files however when I do a search by what we call Cust_id it only pulls out the information from the first line logged for a time stamp. In the above example it would only find Cust_id = 16526719

How can I adjust my query to find the Cust_id per every line that is indexed in Splunk?

Re: Multiple lines with the same time

MarioM — Sun, 26 Feb 2012 10:55:20 GMT

Do you need to keep it as multiline ? because i would get splunk to treat each line as single event and use delims to extract fields.

props.conf:

[your_data_sourcetype]
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)
REPORT-delims=commalist

transforms.conf:

[commalist] 
DELIMS = "," 
FIELDS = field1, field2, field3, ...

Re: Multiple lines with the same time

_d_ — Mon, 27 Feb 2012 00:13:02 GMT

It is more efficient, faster and easier for Splunk to break on every line than to line merge for each event. Try the following stanza on props.conf - it will both break the stream of data at every line AND extract a cust_id field per each event:

[my_sourcetype] SHOULD_LINEMERGE=false LINE_BREAKER=([\r\n]+) EXTRACT-cust_id = (?i)\.\d{3},(?<cust_id>\d+),

Hope this helps.

> please upvote and accept answer if you find it useful - thanks!

Re: Multiple lines with the same time

dobarnes — Mon, 27 Feb 2012 15:14:02 GMT

I tried the both suggestions however I am unable to have Splunk break every line

Re: Multiple lines with the same time

_d_ — Mon, 27 Feb 2012 15:16:06 GMT

dobarnes, did you restart splunk (indexer) after editing line breaking on props.conf?

Re: Multiple lines with the same time

MarioM — Mon, 27 Feb 2012 15:22:31 GMT

and it will only apply to new data

Re: Multiple lines with the same time

dobarnes — Mon, 27 Feb 2012 15:29:28 GMT

I restarted Splunk and ran a real time search to see if the new index data would be line breaked. It did not address the issue.

FYI - I am using a search head and multiple indexer. I have only made the adjustment on the search head. Do I need to do this on the indexers as well?

Re: Multiple lines with the same time

_d_ — Mon, 28 Sep 2020 11:26:59 GMT

YES! The LINE_BREAKER and SHOULD_LINEMERGE settings are effective only at index-time (read indexer) and ignored by the search head :). On the other hand, EXTRACT-cust_id is used by the search head to performs the extraction of the field at search time.

Re: Multiple lines with the same time

dobarnes — Mon, 12 Mar 2012 14:12:35 GMT

Thanks for the answer. It did work when I applied the line break to the indexer.