topic monitor multiple files in folders in Getting Data In

monitor multiple files in folders

ketan_chanana — Thu, 25 Sep 2014 21:55:40 GMT

I want to monitor multiple csv files in a folder name Fwd Test on E drive. I have added below code to my inputs.conf file but Splunk is not picking up data. Do i need to make any changes in props.conf as well?

[monitor://E:\Fwd Test]
index=dir_test
sourcetype=csv

My CSV files in this folder get replaced every 1 hour from Autosys jobs. With this Splunk is duplicating the data indexing. For example: if initially my file report_output has 10 rows initially and after one hour when autosys jobs replace that file with new data for total 20 rows, splunk will index 10+10+20 events. Kindly help

Re: monitor multiple files in folders

jrodman — Fri, 26 Sep 2014 09:52:01 GMT

This question is unclear. You say that Splunk is not picking up data. Then you say that splunk is duplicating the data.
These two statements seem to be in conflict. Can you clarify?

Re: monitor multiple files in folders

srioux — Fri, 26 Sep 2014 12:41:14 GMT

Did you restart Splunk after updating the inputs config?

Re: monitor multiple files in folders

yannK — Fri, 26 Sep 2014 15:28:25 GMT

"Splunk is not picking up data." do you mean that the first version of the file was indexed, but not the next ones ?

search in the _internal splunkd.log for the file name, they may be skipped because considered as duplicates.
Please double check that your new files do not have identical header than the previous one.

and read this documents with care : http://docs.splunk.com/Documentation/Splunk/6.1.3/Data/HowLogFileRotationIsHandled

Re: monitor multiple files in folders

linu1988 — Fri, 26 Sep 2014 15:39:51 GMT

I think the data is now picked from the files but, when we update the existing csv files splunk doesn't take only the updated rows it take the whole CSV content again.. This can be reproduced easily.

Re: monitor multiple files in folders

jrodman — Fri, 26 Sep 2014 15:50:40 GMT

If your application is adding rows to the file by adding bytes to the end, then it is behaving like a logfile, and splunk will handle that.

if your application is adding rows to the file by rewriting the entire file, and the bytes are different, then it is not behaving like a logfile, and splunk is not designed to handle that.