I am having an issue where file names like this "220120808.dat.gz" are not being processed.
After much investigation (manually uncompressing/renaming etc) it turns out that the system props file is identifying the file as a "known_source" source type because of the .dat file extention after the uncompression process.
Seems to be exactly the same issue as per this older thread -> ( http://splunk-base.splunk.com/answers/31415/splunk-not-indexing-modified-files )
My issue is that in my local applications inputs.conf I am defining the sourcetype for all files monitored in a specific directory. The files are being read from that directory so why isn't the sourcetype stanza applied also? At a guess im thinking that the gzip overrides this setting. Yet doesn't apply the app level sourcetype definition afterwards.
So my question is. Why isn't splunk honoring the system>default>local>app>default>local hierachy for source typing?
The suggestion in the older thread is to put in a new source type definition that redefines the dat file for this app.
ie. ../etc/apps/my_app/local/props.conf (below modified from the original system/default/props.conf file)
[source::....(dat)]
sourcetype = my_sourcetype
I've tried this and it doesn't work and still retains the default source type of "known_binary" and thus is not indexed.
My only other way i thought was to just copy that source section and put it in a local system props.conf. My issue then would be that im redefining this source type for ALL .dat files and not just a single application.
More info can be provided.
splunk 4.3.3.
Solution here (only took 2 months 😐 ) : http://splunk-base.splunk.com/answers/60643/archiveprocessor-bypassing-normal-systemlocalpropsconf-p...
As described in the other post putting the sourcetype definition in the local app didn't override the local system setting when the file is inside an archive.
Solution here (only took 2 months 😐 ) : http://splunk-base.splunk.com/answers/60643/archiveprocessor-bypassing-normal-systemlocalpropsconf-p...
As described in the other post putting the sourcetype definition in the local app didn't override the local system setting when the file is inside an archive.
upvote for PYRO
Try moving the sourcetype = my_sourcetype into the /apps/my_app/local/inputs.conf
[monitor:///path/to/*.dat.gz]
whitelist = \d{1,}\.dat\.gz
sourcetype = my_sourcetype
index = my_index
etc...
will you post a snip of the uncompressed .dat and the lines from $SPLUNK/var/log/splunk/splunk.log from where it tries to read the dat.gz?
I added the whitelist as a safety net just in case there was a ton of data in that dir that would regex to *.dat.gz. This being a thing I have learned the hard way.
I re-read your previous answer of "put sourcetype in input.conf". I already had that also (still not working).
That whitelist didn't help.
I put this into the app/local/props.conf.
[source::/path/file(.dat|.dat.gz)]
sourcetype = blah.
Then running ./splunk test sourcetype /path/file.dat.gz shows that it correctly sets the sourcetype to "blah". However the data still doesn't show up inside the index if the file is a .gz. I AM however able to get the .dat file in successfully if I manually uncompress it.
Trying a few other things as I really need to be able to process the logs using only splunk and not an external script to uncompress the files.