Handling text .dat files : How can I override splunks system default props.conf configuration for just a single app?

[Lucas_K]

I am having an issue where file names like this "220120808.dat.gz" are not being processed.

After much investigation (manually uncompressing/renaming etc) it turns out that the system props file is identifying the file as a "known_source" source type because of the .dat file extention after the uncompression process.

Seems to be exactly the same issue as per this older thread -> ( http://splunk-base.splunk.com/answers/31415/splunk-not-indexing-modified-files )

My issue is that in my local applications inputs.conf I am defining the sourcetype for all files monitored in a specific directory. The files are being read from that directory so why isn't the sourcetype stanza applied also? At a guess im thinking that the gzip overrides this setting. Yet doesn't apply the app level sourcetype definition afterwards.

So my question is. Why isn't splunk honoring the system>default>local>app>default>local hierachy for source typing?

The suggestion in the older thread is to put in a new source type definition that redefines the dat file for this app.

ie. ../etc/apps/my_app/local/props.conf (below modified from the original system/default/props.conf file)

[source::....(dat)]
sourcetype = my_sourcetype

I've tried this and it doesn't work and still retains the default source type of "known_binary" and thus is not indexed.

My only other way i thought was to just copy that source section and put it in a local system props.conf. My issue then would be that im redefining this source type for ALL .dat files and not just a single application.

More info can be provided.
splunk 4.3.3.

[Lucas_K · Accepted solution]

Solution here (only took 2 months 😐 ) : http://splunk-base.splunk.com/answers/60643/archiveprocessor-bypassing-normal-systemlocalpropsconf-p...

As described in the other post putting the sourcetype definition in the local app didn't override the local system setting when the file is inside an archive.

[piebob]

upvote for PYRO

[pshumate]

Try moving the sourcetype = my_sourcetype into the /apps/my_app/local/inputs.conf

[monitor:///path/to/*.dat.gz]
whitelist = \d{1,}\.dat\.gz
sourcetype = my_sourcetype
index = my_index
etc...

[pshumate]

will you post a snip of the uncompressed .dat and the lines from $SPLUNK/var/log/splunk/splunk.log from where it tries to read the dat.gz?
I added the whitelist as a safety net just in case there was a ton of data in that dir that would regex to *.dat.gz. This being a thing I have learned the hard way.

[Lucas_K]

I re-read your previous answer of "put sourcetype in input.conf". I already had that also (still not working).

[Lucas_K]

That whitelist didn't help.

I put this into the app/local/props.conf.

[source::/path/file(.dat|.dat.gz)]
sourcetype = blah.

Then running ./splunk test sourcetype /path/file.dat.gz shows that it correctly sets the sourcetype to "blah". However the data still doesn't show up inside the index if the file is a .gz. I AM however able to get the .dat file in successfully if I manually uncompress it.

Trying a few other things as I really need to be able to process the logs using only splunk and not an external script to uncompress the files.