topic Re: How to edit inputs.conf to monitor multiple files with different timestamps from same folder? in Getting Data In

How to edit inputs.conf to monitor multiple files with different timestamps from same folder?

k_harini — Fri, 02 Dec 2016 09:44:08 GMT

I have to monitor 2 files of different source type from same folder with different timestamps continuously for every 15 mins

Both file names start with

sm__nr_ur.csv
sm__tickets.csv

I did this for both files in input.conf

[monitor:///home/gcaaxfer/data/*NR*.csv]
disabled = false
host_segment = 3
index = my_index
sourcetype = my_request

[monitor:///home/gcaaxfer/data/*ticket*.csv]
disabled = false
host_segment = 3
index = my_index
sourcetype = my_request

This is not working. Even if i whitelist with ticket/.csv for that monitor, it doesn't seem to work. Any idea how to get this to working? Please help with your inputs. Thanks!

I have used <*> symbol.. but its not getting displayed

Re: How to edit inputs.conf to monitor multiple files with different timestamps from same folder?

aaraneta_splunk — Tue, 29 Sep 2020 11:58:30 GMT

@k_harini - Just so you know, there is special markup language on this site so certain symbols will transform your post. If you wrap a word in the asterisk symbol * or _, without wrapping it in a code sample, it will italicize the word. If you wish to show the * (i.e. you are displaying sample code), simply click on the Code Sample icon to the right of the Blockquote icon in the formatting toolbar. That is how I was able to edit your post so that the * will display.

Re: How to edit inputs.conf to monitor multiple files with different timestamps from same folder?

lguinn2 — Sat, 03 Dec 2016 01:28:29 GMT

Here are a few questions that may help:

Are these files on the same machine as the Splunk indexer?
What user is running Splunk? Does that user have permission to access the /home/gcaaxfer/data directory and the files?
Splunk is case-sensitive. In your inputs.conf you have *NR* but the nr is lower-case in the file name you showed. Are your stanzas correct with regards to upper/lower case?
Does my_index exist?
Are you checking for your data by searching with index=my_index specifically? Are you sure that your search is covering the proper time range? Try searching "all-time".

You should not need to use a whitelist to get these files.
You can also take a look at the Splunk internal log, splunkd.log, to check for errors. You can find it at $SPLUNK_HOME/var/log/splunk/splunkd.log
You should see messages when Splunk starts, including messages telling what files it is indexing - and if it is not able to the index the files, there should be error messages.

Re: How to edit inputs.conf to monitor multiple files with different timestamps from same folder?

k_harini — Sat, 03 Dec 2016 05:59:26 GMT

Thanks for your response..
1. Yes same machine.. no forwarder here..
2. Users have access to the folder and files are transferred here through sftp
3. That is correct.. its NR (both file name and monitor)
4. Yes index exist.. and both have different indexes

If i do one time indexing it works fine.. on adding monitor to the file .. its not getting indexed.. should i add crcsalt flag as all files start with SM..and some timestamp.. will it work..? I will check the logs too... Thank you..

Re: How to edit inputs.conf to monitor multiple files with different timestamps from same folder?

k_harini — Sat, 03 Dec 2016 06:01:46 GMT

ok thanks...

Re: How to edit inputs.conf to monitor multiple files with different timestamps from same folder?

ddrillic — Sat, 03 Dec 2016 07:11:59 GMT

Check please the splunkd.log ...

Re: How to edit inputs.conf to monitor multiple files with different timestamps from same folder?

k_harini — Mon, 05 Dec 2016 06:39:53 GMT

I checked logs.. I get Filewatcher: Error getting attributes from input path wild card.. Is there anything wrong with my input path with wildcard.. I too suspect that.. Is my input path valid..what change should i make in path to pick up correct file..? please let me know.

Re: How to edit inputs.conf to monitor multiple files with different timestamps from same folder?

gooza — Mon, 05 Dec 2016 08:23:43 GMT

I agree with lguinn , white listing will solve the problem ( it did for me , the only thing I need to make sure is that the monitored folder is a little bit different

for example - input.conf:

 [monitor:///home/gcaaxfer/data/]
 disabled = false
 host_segment = 3
 index = my_index
 sourcetype = my_request
 whitelist = (sm_nr_.*\.csv)  
 crcSalt = <SOURCE>

 [monitor:///home/gcaaxfer/data]
 disabled = false
 host_segment = 3
 index = my_index
 sourcetype = my_request
 whitelist =(sm_.*tickets\.csv)
 crcSalt = <SOURCE>

notice that one of the monitored path has '/' at the end and one without.

this enabled me to monitor easily different files in the same directory

Re: How to edit inputs.conf to monitor multiple files with different timestamps from same folder?

svaleson — Tue, 29 Sep 2020 14:07:04 GMT

The issue here is that you have overlapping monitor stanzas. The * in the source implicitly tells Splunk to monitor files in /home/gcaaxfer/data/ for both, resulting in the overlapping stanzas and only one of them being read.

The recommended approach to solve this issue is to create one stanza in inputs.conf to read in both sets of files and also deploy a props.conf within the same add-on on the Forwarder to specify the sourcetype based on the source. As follows:

inputs.conf:
[monitor:///home/gcaaxfer/data]
whitelist = (*NR*.csv | *ticket*.csv)
host_segment = 3
index = my_index

disabled = false

props.conf:
[source::.../*NR*.csv]
sourcetype = request_nr

[source::.../*ticket*.csv]

sourcetype = request_ticket

Now that you've assigned them to separate sourcetypes, you can easily update your props.conf on your Indexers and/or Heavy Forwarders to properly perform the separate time extractions.