How to monitor CSV files with inputs.conf?

[Chandras11]

Hi Everyone,

I recently read about the splunk file monitoring with inputs.conf at http://docs.splunk.com/Documentation/SplunkCloud/7.0.3/Data/Monitorfilesanddirectorieswithinputs.con... .
I am using Splunk Enterprise 7 local instance and want to upload 2 CSV files with same index but different sourcetype to splunk daily.
Therefore: I did the following steps:-
1. Create C:\SaeXXXLog Folder and put the Creator_29_05.csv and Receiver_29_05.csv at this location (only 2 files).
2. Change the inputs.conf at C:\Program Files\Splunk\etc\system\local as following:

[monitor://C:\SaeXXXLog]
whitelist = Creator*.csv
index = Index1
sourcetype = Creator
ignoreOlderThan = 1d

[monitor://C:\SaeXXXLog]
index = Index1
whitelist = Receiver*.csv
sourcetype = Receiver
ignoreOlderThan = 1d

However, when I try to check for the events in Splunk, I don't get any event uploaded there in last 24 hours. I am not sure if I need to make any other changes in some other file or the code written by me is wrong for it.

I am looking forward to your expert opinion.

[FrankVl · Accepted solution]

Looking at it once more, I think the whitelist is defined incorrectly. That setting is supposed to take a regex, not a string with wildcards like *.

Probably easier to just include the filename in the monitor path, like so: [monitor://C:\SaeXXXLog\Creator*.csv]

Or define your whitelist as a proper regex. Note the whitelist regex is evaluated against the full path, but as long as it matches part of it, it will trigger. So you can keep it as simple as whitelist = Creator to match files that have the string 'Creator' somewhere in their path / filename.

[Chandras11]

change and restarted, but no new events or the events with old timestamp. Do you know, where can I see the logs for this process in splunk.
Thanks for your help.

[FrankVl]

$SPLUNK_HOME/var/log/splunk/splunkd.log should show that it starts monitoring the folder and then should mention it detected some file etc.

[Chandras11]

wow.. thanks for the info. Now there are some warning as:
05-30-2018 14:23:01.209 +0200 WARN DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\scheduler_nobody_YWlhbS1pdHNtLXRpY2tldGFuYWx5c2lzLXVpLWNvbW1vbi1uZXc_RMD5a9444e1e3ed8f2c8_at_1527613200_214\metadata.csv

05-30-2018 14:23:01.210 +0200 WARN DispatchReaper - Failed to read search info for id=scheduler_nobody_YWlhbS1pdHNtLXRpY2tldGFuYWx5c2lzLXVpLWNvbW1vbi1uZXc_RMD5a9444e1e3ed8f2c8_at_1527613500_218

05-30-2018 14:23:01.211 +0200 WARN DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\scheduler_nobody_YWlhbS1pdHNtLXRpY2tldGFuYWx5c2lzLXVpLWNvbW1vbi1uZXc_RMD5a9444e1e3ed8f2c8_at_1527613500_218\metadata.csv

[dflodstrom]

Try using grep (or SplunK! index=_internal) and search for your file name to see if you see a message where Splunk has recognized your input and will start monitoring your file/path.

[Chandras11]

finally, its started working. I really dont know how but I am having the data in my index as of now.

Thanks a lot for this help.

As a matter of fact, I didnot use the web UI directly. I was thinking that its possible only with the input.conf... how foolish am I. 🙂

Its working now 🙂

[FrankVl]

Not sure what that is about, but doesn't look too relevant to those monitor inputs. You can look in the log around the time of restarting it, there you should be able to find events related to activating the monitor input etc.

[FrankVl]

Did you restart splunk after making those inputs.conf changes? Have you checked in splunkd.log whether there are any messages related to those inputs? Have you tried without the ignoreOlderThan = 1d?

[Chandras11]

i restarted it multiple times.. i havent removed the ignoreOlderThan = 1d, which I am doing now. Does I need to do something like
[monitor://C:\SaeXXXLog\Receiver]
whitelist = *.csv$

just read it on https://answers.splunk.com/answers/238359/why-does-inputsconf-does-not-respect-the-use-of-a.html

[Chandras11]

removed ignoreOlderThen as well. Is there any complete documentation where we know what to change exactly. I am still facing the same issue

[FrankVl]

See my answer below, looking at it again, I think your way of using whitelist is incorrect.

[dflodstrom]

If you're monitoring a CSV file that doesn't have a timestamp column it is possible that your events are being timestamped incorrectly. If you're able, try searching a much broader time range to see if anything from your CSV is being indexed?

[Chandras11]

Hi, Thanks a lot for your response.
Yes the CSV doesn't have a time stamp and therefore, I always use to put the auto as timestamp while uploading the data manually.. let me check for that if I have events from today in this CSV. Is there any other possibility of error here?

[dflodstrom]

You'll need to specify the timestamp in your monitor input just as you do when you input manually through the web UI. If you're not finding the data check that your input and whitelist conform with what is expected as described here --> https://docs.splunk.com/Documentation/Splunk/7.1.0/Data/Specifyinputpathswithwildcards

[Chandras11]

Hi, Sorry to disturb you again, "You'll need to specify the timestamp in your monitor input" : I don't find any command in the shared link. Am i missing something here.
My apologies but I have just started working with input.conf today and there are few things which I am not able to comprehend properly.

[dflodstrom]

Sorry, that configuration will go in props.conf http://docs.splunk.com/Documentation/Splunk/7.1.0/Data/Configuretimestamprecognition

You can use something like DATETIME_CONFIG = current if you want the events to be timestamped with current time as they're ingested

[Chandras11]

I created a prop conf but it is still not working. I also tried to check events with the new file and they are not indexed in Splunk

[Chandras11]

http://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf : it says that I can modify the Time stamp in the Propsconf file. I am checking there further