i am indexing .dat file which contains more than 5000 events. in the middle 1 or 2 events breaked wrongly This the config i used Props.conf NO_BINARY_CHECK = true BREAK_ONLY_BEFORE = ^\d{1,11}\s?,(([^\,]+)?\,?.?),(([^\,]+)?\,?.?) MAX_TIMESTAMP_LOOKAHEAD = 100 TIME_FORMAT = %Y%m%d%H%M%S%6N TIME_PREFIX = ^(?:[^,\n]*,){7} disabled = false pulldown_type = true inputs.conf [monitor:///xxxx] disabled = false whitelist=*.dat time_before_close = 120 multiline_event_extra_waittime = true index = xxxx sourcetype = yyyy Actual Events 00000000000,,xxxx,40673673,19.08.2016,14:00,21:00,20160818070100184759,/ablive/data/yyyy/serial/yyyy/DISTRIBUTION/DELIVERY/delivery_messages_inbound/pending/./xxxx201608180700060000.csv,xxxx201608180700060000.csv,26,c2038af5-5b95-4532-bfa2-e2fa54d8a29e,22a301ea-26-a666-5e1b87780-ac168f26_57b54f17_2dc00d6-11b7,22a301ea-26-a666-5e1b87780-ac168f26_57b54f17_2dc00d6-1232,2016-08-18T07:01:50.679Z,2016-08-18T07:01:52.994Z,44,GB,Scheduled,Success,SUCCESS,SUCCESS 00000000000,,xxxx,40667760,19.08.2016,17:00,21:00,20160818070100167747,/ablive/data/yyyy/serial/yyyy/DISTRIBUTION/DELIVERY/delivery_messages_inbound/pending/./xxxx201608180700060000.csv,xxxx201608180700060000.csv,24,854f6e61-bf00-4914-9799-c539eb30be81,22a301ea-26-a666-5e1b87780-ac168f26_57b54f17_2dc00d6-1023,22a301ea-26-a666-5e1b87780-ac168f26_57b54f17_2dc00d6-1066,2016-08-18T07:01:46.089Z,2016-08-18T07:01:49.160Z,44,GB,Scheduled,Success,SUCCESS,SUCCESS Indexed Events e,22a301ea-26-a666-5e1b87780-ac168f26_57b54f17_2dc00d6-11b7,22a301ea-26-a666-5e1b87780-ac168f26_57b54f17_2dc00d6-1232,2016-08-18T07:01:50.679Z,2016-08-18T07:01:52.994Z,44,GB,Scheduled,Success,SUCCESS,SUCCESS 60,19.08.2016,17:00,21:00,20160818070100167747,/ablive/data/yyyy/serial/yyyy/DISTRIBUTION/DELIVERY/delivery_messages_inbound/pending/./xxxx201608180700060000.csv,xxxx201608180700060000.csv,24,854f6e61-bf00-4914-9799-c539eb30be81,22a301ea-26-a666-5e1b87780-ac168f26_57b54f17_2dc00d6-1023,22a301ea-26-a666-5e1b87780-ac168f26_57b54f17_2dc00d6-1066,2016-08-18T07:01:46.089Z,2016-08-18T07:01:49.160Z,44,GB,Scheduled,Success,SUCCESS,SUCCESS 00000000000,,xxxx,40673673,19.08.2016,14:00,21:00,20160818070100184759,/ablive/data/yyyy/serial/yyyy/DISTRIBUTION/DELIVERY/delivery_messages_inbound/pending/./xxxx201608180700060000.csv,xxxx201608180700060000.csv,26,c2038af5-5b95-4532-bfa2-e2fa54d8a29 00000000000,,xxxx,406677 Indextimings indextime source count 2016-08-18 07:01:49 xxxx 2162 2016-08-18 07:01:52 xxxx 2 2016-08-18 07:01:53 xxxx 2137 2016-08-18 07:01:56 xxxx 2 2016-08-18 07:01:58 xxxx 1266 same file indexed in above mentioned time and count 2 contains splitted events. I used time_before_close and multiline_event_extra_waittime=true even though 1 or 2 events get splitted. Thanks in advance. ... View more

I am using the following configuration in props.conf. It is splitting most of the events correctly, but 2 or 3 events are collapsed. Should I need to include SHOULD_LINEMERGE = false ? [my_source_type] NO_BINARY_CHECK = true BREAK_ONLY_BEFORE = ^\d{1,11}\s?,(([^\,]+)?\,?\.?),(([^\,]+)?\,?\.?) MAX_TIMESTAMP_LOOKAHEAD = 100 TIME_FORMAT = %Y%m%d%H%M%S%6N TIME_PREFIX = ^(?:[^,\n]*,){7} disabled = false pulldown_type = true This is a .dat file and it has more than 8000 events on a single file. Sample data Actual events 07986376244,Mrs,xxxx,40369036,29.06.2016,14:00,21:00,20160628070106529271,/ablive/data/xx/serial/yy/DISTRIBUTION/DELIVERY/delivery_messages_inbound/pending/./MessageReminderPM201606280700120000.csv,MessageReminderPM201606280700120000.csv,38,4c7ca670-eddf-4362-8f4b-20ea99007a0b,225b00fe-26-a633-5e21f14e2-ac168f26_5772129e_37501fc-1b5a,225b00fe-26-a633-5e21f14e2-ac168f26_5772129e_37501fc-1bca,2016-06-28T07:02:23.224Z,2016-06-28T07:02:26.890Z,44,GB,Scheduled,Success,SUCCESS,SUCCESS 07941158158,Mr,yyyyy,40360516,29.06.2016,14:00,21:00,20160628070106516893,/ablive/data/xx/serial/yy/DISTRIBUTION/DELIVERY/delivery_messages_inbound/pending/./MessageReminderPM201606280700120000.csv,MessageReminderPM201606280700120000.csv,36,4a140e0f-69e4-44d3-a5ce-dfb186c9a081,225b00fe-26-a633-5e21f14e2-ac168f26_5772129e_37501fc-19c6,225b00fe-26-a633-5e21f14e2-ac168f26_5772129e_37501fc-1a2f,2016-06-28T07:02:17.050Z,2016-06-28T07:02:19.816Z,44,GB,Scheduled,Success,SUCCESS,SUCCESS indexed events ELIVERY/delivery_messages_inbound/pending/./MessageReminderPM201606280700120000.csv,MessageReminderPM201606280700120000.csv,38,4c7ca670-eddf-4362-8f4b-20ea99007a0b,225b00fe-26-a633-5e21f14e2-ac168f26_5772129e_37501fc-1b5a,225b00fe-26-a633-5e21f14e2-ac168f26_5772129e_37501fc-1bca,2016-06-28T07:02:23.224Z,2016-06-28T07:02:26.890Z,44,GB,Scheduled,Success,SUCCESS,SUCCESS xx/serial/JL/DISTRIBUTION/DELIVERY/delivery_messages_inbound/pending/./MessageReminderPM201606280700120000.csv,MessageReminderPM201606280700120000.csv,36,4a140e0f-69e4-44d3-a5ce-dfb186c9a081,225b00fe-26-a633-5e21f14e2-ac168f26_5772129e_37501fc-19c6,225b00fe-26-a633-5e21f14e2-ac168f26_5772129e_37501fc-1a2f,2016-06-28T07:02:17.050Z,2016-06-28T07:02:19.816Z,44,GB,Scheduled,Success,SUCCESS,SUCCESS 07986356244,Mrs,Mason,40369036,29.06.2016,14:00,21:00,20160628070106529271,/ablive/data/xx/serial/yy/DISTRIBUTION/D 07941156158,Mr,Hurley,40360516,29.06.2016,14:00,21:00,20160628070106516893,/ablive/data/ Thanks in advance ... View more