All Apps and Add-ons

How to monitor CSV files with inputs.conf?

Chandras11
Communicator

Hi Everyone,

I recently read about the splunk file monitoring with inputs.conf at http://docs.splunk.com/Documentation/SplunkCloud/7.0.3/Data/Monitorfilesanddirectorieswithinputs.con... .
I am using Splunk Enterprise 7 local instance and want to upload 2 CSV files with same index but different sourcetype to splunk daily.
Therefore: I did the following steps:-
1. Create C:\SaeXXXLog Folder and put the Creator_29_05.csv and Receiver_29_05.csv at this location (only 2 files).
2. Change the inputs.conf at C:\Program Files\Splunk\etc\system\local as following:

[monitor://C:\SaeXXXLog]
whitelist = Creator*.csv
index = Index1
sourcetype = Creator
ignoreOlderThan = 1d

[monitor://C:\SaeXXXLog]
index = Index1
whitelist = Receiver*.csv
sourcetype = Receiver
ignoreOlderThan = 1d

However, when I try to check for the events in Splunk, I don't get any event uploaded there in last 24 hours. I am not sure if I need to make any other changes in some other file or the code written by me is wrong for it.

I am looking forward to your expert opinion.

0 Karma
1 Solution

FrankVl
Ultra Champion

Looking at it once more, I think the whitelist is defined incorrectly. That setting is supposed to take a regex, not a string with wildcards like *.

Probably easier to just include the filename in the monitor path, like so: [monitor://C:\SaeXXXLog\Creator*.csv]

Or define your whitelist as a proper regex. Note the whitelist regex is evaluated against the full path, but as long as it matches part of it, it will trigger. So you can keep it as simple as whitelist = Creator to match files that have the string 'Creator' somewhere in their path / filename.

View solution in original post

FrankVl
Ultra Champion

Looking at it once more, I think the whitelist is defined incorrectly. That setting is supposed to take a regex, not a string with wildcards like *.

Probably easier to just include the filename in the monitor path, like so: [monitor://C:\SaeXXXLog\Creator*.csv]

Or define your whitelist as a proper regex. Note the whitelist regex is evaluated against the full path, but as long as it matches part of it, it will trigger. So you can keep it as simple as whitelist = Creator to match files that have the string 'Creator' somewhere in their path / filename.

View solution in original post

Chandras11
Communicator

change and restarted, but no new events or the events with old timestamp. Do you know, where can I see the logs for this process in splunk.
Thanks for your help.

0 Karma

FrankVl
Ultra Champion

$SPLUNK_HOME/var/log/splunk/splunkd.log should show that it starts monitoring the folder and then should mention it detected some file etc.

Chandras11
Communicator

wow.. thanks for the info. Now there are some warning as:
05-30-2018 14:23:01.209 +0200 WARN DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\scheduler_nobody_YWlhbS1pdHNtLXRpY2tldGFuYWx5c2lzLXVpLWNvbW1vbi1uZXc_RMD5a9444e1e3ed8f2c8_at_1527613200_214\metadata.csv

05-30-2018 14:23:01.210 +0200 WARN DispatchReaper - Failed to read search info for id=scheduler_nobody_YWlhbS1pdHNtLXRpY2tldGFuYWx5c2lzLXVpLWNvbW1vbi1uZXc_RMD5a9444e1e3ed8f2c8_at_1527613500_218

05-30-2018 14:23:01.211 +0200 WARN DispatchSearchMetadata - could not read metadata file: C:\Program Files\Splunk\var\run\splunk\dispatch\scheduler_nobody_YWlhbS1pdHNtLXRpY2tldGFuYWx5c2lzLXVpLWNvbW1vbi1uZXc_RMD5a9444e1e3ed8f2c8_at_1527613500_218\metadata.csv

0 Karma

dflodstrom
Builder

Try using grep (or SplunK! index=_internal) and search for your file name to see if you see a message where Splunk has recognized your input and will start monitoring your file/path.

0 Karma

Chandras11
Communicator

finally, its started working. I really dont know how but I am having the data in my index as of now.

Thanks a lot for this help.

As a matter of fact, I didnot use the web UI directly. I was thinking that its possible only with the input.conf... how foolish am I. 🙂

Its working now 🙂

FrankVl
Ultra Champion

Not sure what that is about, but doesn't look too relevant to those monitor inputs. You can look in the log around the time of restarting it, there you should be able to find events related to activating the monitor input etc.

FrankVl
Ultra Champion

Did you restart splunk after making those inputs.conf changes? Have you checked in splunkd.log whether there are any messages related to those inputs? Have you tried without the ignoreOlderThan = 1d?

Chandras11
Communicator

i restarted it multiple times.. i havent removed the ignoreOlderThan = 1d, which I am doing now. Does I need to do something like
[monitor://C:\SaeXXXLog\Receiver]
whitelist = *.csv$

just read it on https://answers.splunk.com/answers/238359/why-does-inputsconf-does-not-respect-the-use-of-a.html

0 Karma

Chandras11
Communicator

removed ignoreOlderThen as well. Is there any complete documentation where we know what to change exactly. I am still facing the same issue

0 Karma

FrankVl
Ultra Champion

See my answer below, looking at it again, I think your way of using whitelist is incorrect.

0 Karma

dflodstrom
Builder

If you're monitoring a CSV file that doesn't have a timestamp column it is possible that your events are being timestamped incorrectly. If you're able, try searching a much broader time range to see if anything from your CSV is being indexed?

Chandras11
Communicator

Hi, Thanks a lot for your response.
Yes the CSV doesn't have a time stamp and therefore, I always use to put the auto as timestamp while uploading the data manually.. let me check for that if I have events from today in this CSV. Is there any other possibility of error here?

0 Karma

dflodstrom
Builder

You'll need to specify the timestamp in your monitor input just as you do when you input manually through the web UI. If you're not finding the data check that your input and whitelist conform with what is expected as described here --> https://docs.splunk.com/Documentation/Splunk/7.1.0/Data/Specifyinputpathswithwildcards

0 Karma

Chandras11
Communicator

Hi, Sorry to disturb you again, "You'll need to specify the timestamp in your monitor input" : I don't find any command in the shared link. Am i missing something here.
My apologies but I have just started working with input.conf today and there are few things which I am not able to comprehend properly.

0 Karma

dflodstrom
Builder

Sorry, that configuration will go in props.conf http://docs.splunk.com/Documentation/Splunk/7.1.0/Data/Configuretimestamprecognition

You can use something like DATETIME_CONFIG = current if you want the events to be timestamped with current time as they're ingested

Chandras11
Communicator

I created a prop conf but it is still not working. I also tried to check events with the new file and they are not indexed in Splunk

0 Karma

Chandras11
Communicator

http://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf : it says that I can modify the Time stamp in the Propsconf file. I am checking there further

0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!