https://community.splunk.com/t5/Getting-Data-In/Why-is-some-data-being-indexed-as-separate-entries-while/m-p/170489#M34469 <P>I don't think this is a bug. I would avoid locking the file in general, as it may have unexpected performance impacts.</P> <P>What are the settings in the <CODE>inputs.conf</CODE> stanza that is monitoring the output folder? I would suggest this:</P> <PRE><CODE>[monitor://yourdirectorypathhere] index = theindexname sourcetype = csv ignoreOlderThan = 30d </CODE></PRE> <P>Or, you might find a pretrained sourcetype that better fits your data here: <A href="http://docs.splunk.com/Documentation/Splunk/6.2.1/Data/Listofpretrainedsourcetypes">List of pretrained sourcetypes</A><BR /> If you are not cleaning out the older files (which you should), the <CODE>ignoreOlderThan</CODE> will help Splunk's performance if the directory becomes full of older files that have already been indexed and that will never be updated.</P> <P>If you don't want to use the <CODE>csv</CODE> sourcetype, you may need to place a <CODE>props.conf</CODE> file on your indexer that explicitly sets the parsing rules for the sourcetype that you choose.</P> Mon, 05 Jan 2015 00:11:14 GMT lguinn2 2015-01-05T00:11:14Z https://community.splunk.com/t5/Getting-Data-In/Why-is-some-data-being-indexed-as-separate-entries-while/m-p/170488#M34468 <P>I have a console app which reads data from table storage, and writes it out onto a csv file. I monitor each of the output folder, I checked to see if the data is being uploaded properly which it does. However there is disconnected data inside the index when I look at all the records. Reason I know is because the records in the csv file doesn't match the records on splunk.</P> <P>An entry contains 18 fields some of the entries are being split in the middle of the entry. <BR /> Example Entry<BR /> 12/15/2015, Name, ID, Number, Guid ....</P> <P>Splunk logs it as 2 seperate entry</P> <P>12/15/2015,Name,ID,<BR /> Number,Guid ...</P> <P>The console app runs every day once at midnight and not all entries are being malformed just a few of them. The strange thing is that the data is still all there it just some of them are separated. Anyone know of a work around or is this a bug? <BR /> I was thinking of locking the file until it's finish writing but I'm not sure how splunk would react to fileshare locking. </P> Mon, 29 Dec 2014 22:20:14 GMT https://community.splunk.com/t5/Getting-Data-In/Why-is-some-data-being-indexed-as-separate-entries-while/m-p/170488#M34468 chatterjb 2014-12-29T22:20:14Z https://community.splunk.com/t5/Getting-Data-In/Why-is-some-data-being-indexed-as-separate-entries-while/m-p/170489#M34469 <P>I don't think this is a bug. I would avoid locking the file in general, as it may have unexpected performance impacts.</P> <P>What are the settings in the <CODE>inputs.conf</CODE> stanza that is monitoring the output folder? I would suggest this:</P> <PRE><CODE>[monitor://yourdirectorypathhere] index = theindexname sourcetype = csv ignoreOlderThan = 30d </CODE></PRE> <P>Or, you might find a pretrained sourcetype that better fits your data here: <A href="http://docs.splunk.com/Documentation/Splunk/6.2.1/Data/Listofpretrainedsourcetypes">List of pretrained sourcetypes</A><BR /> If you are not cleaning out the older files (which you should), the <CODE>ignoreOlderThan</CODE> will help Splunk's performance if the directory becomes full of older files that have already been indexed and that will never be updated.</P> <P>If you don't want to use the <CODE>csv</CODE> sourcetype, you may need to place a <CODE>props.conf</CODE> file on your indexer that explicitly sets the parsing rules for the sourcetype that you choose.</P> Mon, 05 Jan 2015 00:11:14 GMT https://community.splunk.com/t5/Getting-Data-In/Why-is-some-data-being-indexed-as-separate-entries-while/m-p/170489#M34469 lguinn2 2015-01-05T00:11:14Z