How to break single events created from multiple events

[nirmalya2006]

Hi All

So I have a batch job that creates flat files per day. Each line in the flat files is an individual event.
The flat files are forwarded to Splunk using Universal forwarders.

Now Splunk has somehow managed to bundle some individual events into one single event.
This is completely random and at the choice of our dear splunk which all events it wanted to bundle.

Since this has happened, my searches do not return those events that are bundled inside the single event though each event has its own date and time.

Can anyone please help me get the events broken back to being distinct individual events so that I can query all events.
Below is how my events are being displayed.

6/19/17
5:00:00.000 AM  
... 2 lines omitted ...
2017-06-19 02:00:00|SLC1|API|TRUE|46.82|0.05|0.00|0.00
2017-06-19 03:00:00|SLC1|API|TRUE|46.82|0.05|0.00|0.00
2017-06-19 04:00:00|SLC1|API|TRUE|43.59|0.05|0.00|0.00
... 21 lines omitted ...
2017-06-19 06:00:00|SLC1|API|TRUE|5567.02|6.46|0.00|0.00
... 11 lines omitted ...
2017-06-19 15:00:00|SLC1|API|TRUE|502.20|0.58|0.00|0.00
Show all 50 lines
host = 10.40.10.89 source = /reporting/MIPS_SLC1_MIPS_METRICS_20170619.txt sourcetype = mips_reports
6/19/17
4:00:00.000 AM  
2017-06-18 23:00:00|SLC1|API|TRUE|36.33|0.04|0.00|0.00
host = 10.40.10.89 source = /reporting/MIPS_SLC1_MIPS_METRICS_20170618.txt sourcetype = mips_reports
6/19/17
4:00:00.000 AM  
2017-06-18 23:00:00|SLC1|API|API|14332.96|16.65|0.00|0.00
host = 10.40.10.89 source = /reporting/MIPS_SLC1_MIPS_METRICS_20170618.txt sourcetype = mips_reports

[niketn · Accepted solution]

@nirmalya2006, if you want event breaks on every line then you should define the following for your sourcetype in props.conf file:

[YourSourceTypeName]
SHOULD_LINEMERGE=false

http://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf#Line_breaking

You might have to re-index your original data as well. Please sample your events and test out sourcetype definition very carefully in non prod Splunk instance, so that Splunk is able to identify event timestamp and event data properly.
Following is the step by step documentation for defining sourcetype through Splunk Web:
http://docs.splunk.com/Documentation/Splunk/latest/Data/Setsourcetype