I have to monitor 2 files of different source type from same folder with different timestamps continuously for every 15 mins
Both file names start with
sm__nr_ur.csv
sm__tickets.csv
I did this for both files in input.conf
[monitor:///home/gcaaxfer/data/*NR*.csv]
disabled = false
host_segment = 3
index = my_index
sourcetype = my_request
[monitor:///home/gcaaxfer/data/*ticket*.csv]
disabled = false
host_segment = 3
index = my_index
sourcetype = my_request
This is not working. Even if i whitelist with ticket/.csv for that monitor, it doesn't seem to work. Any idea how to get this to working? Please help with your inputs. Thanks!
I have used <*> symbol.. but its not getting displayed
I agree with lguinn , white listing will solve the problem ( it did for me , the only thing I need to make sure is that the monitored folder is a little bit different
for example - input.conf:
[monitor:///home/gcaaxfer/data/]
disabled = false
host_segment = 3
index = my_index
sourcetype = my_request
whitelist = (sm_nr_.*\.csv)
crcSalt = <SOURCE>
[monitor:///home/gcaaxfer/data]
disabled = false
host_segment = 3
index = my_index
sourcetype = my_request
whitelist =(sm_.*tickets\.csv)
crcSalt = <SOURCE>
notice that one of the monitored path has '/' at the end and one without.
this enabled me to monitor easily different files in the same directory
The issue here is that you have overlapping monitor stanzas. The * in the source implicitly tells Splunk to monitor files in /home/gcaaxfer/data/ for both, resulting in the overlapping stanzas and only one of them being read.
The recommended approach to solve this issue is to create one stanza in inputs.conf to read in both sets of files and also deploy a props.conf within the same add-on on the Forwarder to specify the sourcetype based on the source. As follows:
inputs.conf:
[monitor:///home/gcaaxfer/data]
whitelist = (*NR*.csv | *ticket*.csv)
host_segment = 3
index = my_index
props.conf:
[source::.../*NR*.csv]
sourcetype = request_nr
[source::.../*ticket*.csv]
Now that you've assigned them to separate sourcetypes, you can easily update your props.conf on your Indexers and/or Heavy Forwarders to properly perform the separate time extractions.
I agree with lguinn , white listing will solve the problem ( it did for me , the only thing I need to make sure is that the monitored folder is a little bit different
for example - input.conf:
[monitor:///home/gcaaxfer/data/]
disabled = false
host_segment = 3
index = my_index
sourcetype = my_request
whitelist = (sm_nr_.*\.csv)
crcSalt = <SOURCE>
[monitor:///home/gcaaxfer/data]
disabled = false
host_segment = 3
index = my_index
sourcetype = my_request
whitelist =(sm_.*tickets\.csv)
crcSalt = <SOURCE>
notice that one of the monitored path has '/' at the end and one without.
this enabled me to monitor easily different files in the same directory
Here are a few questions that may help:
/home/gcaaxfer/data
directory and the files?*NR*
but the nr
is lower-case in the file name you showed. Are your stanzas correct with regards to upper/lower case?index=my_index
specifically? Are you sure that your search is covering the proper time range? Try searching "all-time".You should not need to use a whitelist to get these files.
You can also take a look at the Splunk internal log, splunkd.log, to check for errors. You can find it at $SPLUNK_HOME/var/log/splunk/splunkd.log
You should see messages when Splunk starts, including messages telling what files it is indexing - and if it is not able to the index the files, there should be error messages.
I checked logs.. I get Filewatcher: Error getting attributes from input path wild card.. Is there anything wrong with my input path with wildcard.. I too suspect that.. Is my input path valid..what change should i make in path to pick up correct file..? please let me know.
Thanks for your response..
1. Yes same machine.. no forwarder here..
2. Users have access to the folder and files are transferred here through sftp
3. That is correct.. its NR (both file name and monitor)
4. Yes index exist.. and both have different indexes
If i do one time indexing it works fine.. on adding monitor to the file .. its not getting indexed.. should i add crcsalt flag as all files start with SM..and some timestamp.. will it work..? I will check the logs too... Thank you..
Check please the splunkd.log
...
@k_harini - Just so you know, there is special markup language on this site so certain symbols will transform your post. If you wrap a word in the asterisk symbol *
or _
, without wrapping it in a code sample
, it will italicize the word. If you wish to show the *
(i.e. you are displaying sample code), simply click on the Code Sample icon to the right of the Blockquote icon in the formatting toolbar. That is how I was able to edit your post so that the *
will display.
ok thanks...