Getting Data In
Provide Splunk Cloud feedback in this confidential UX survey by June 17
for a chance to win a $200 Amazon gift card!

Handling text .dat files : How can I override splunks system default props.conf configuration for just a single app?

Lucas_K
Motivator

I am having an issue where file names like this "220120808.dat.gz" are not being processed.

After much investigation (manually uncompressing/renaming etc) it turns out that the system props file is identifying the file as a "known_source" source type because of the .dat file extention after the uncompression process.

Seems to be exactly the same issue as per this older thread -> ( http://splunk-base.splunk.com/answers/31415/splunk-not-indexing-modified-files )

My issue is that in my local applications inputs.conf I am defining the sourcetype for all files monitored in a specific directory. The files are being read from that directory so why isn't the sourcetype stanza applied also? At a guess im thinking that the gzip overrides this setting. Yet doesn't apply the app level sourcetype definition afterwards.

So my question is. Why isn't splunk honoring the system>default>local>app>default>local hierachy for source typing?

The suggestion in the older thread is to put in a new source type definition that redefines the dat file for this app.

ie. ../etc/apps/my_app/local/props.conf (below modified from the original system/default/props.conf file)

[source::....(dat)]
sourcetype = my_sourcetype

I've tried this and it doesn't work and still retains the default source type of "known_binary" and thus is not indexed.

My only other way i thought was to just copy that source section and put it in a local system props.conf. My issue then would be that im redefining this source type for ALL .dat files and not just a single application.

More info can be provided.
splunk 4.3.3.

0 Karma
1 Solution

Lucas_K
Motivator

Solution here (only took 2 months 😐 ) : http://splunk-base.splunk.com/answers/60643/archiveprocessor-bypassing-normal-systemlocalpropsconf-p...

As described in the other post putting the sourcetype definition in the local app didn't override the local system setting when the file is inside an archive.

View solution in original post

Lucas_K
Motivator

Solution here (only took 2 months 😐 ) : http://splunk-base.splunk.com/answers/60643/archiveprocessor-bypassing-normal-systemlocalpropsconf-p...

As described in the other post putting the sourcetype definition in the local app didn't override the local system setting when the file is inside an archive.

View solution in original post

piebob
Motivator

upvote for PYRO

0 Karma

pshumate
Explorer

Try moving the sourcetype = my_sourcetype into the /apps/my_app/local/inputs.conf

[monitor:///path/to/*.dat.gz]
whitelist = \d{1,}\.dat\.gz
sourcetype = my_sourcetype
index = my_index
etc...
0 Karma

pshumate
Explorer

will you post a snip of the uncompressed .dat and the lines from $SPLUNK/var/log/splunk/splunk.log from where it tries to read the dat.gz?
I added the whitelist as a safety net just in case there was a ton of data in that dir that would regex to *.dat.gz. This being a thing I have learned the hard way.

0 Karma

Lucas_K
Motivator

I re-read your previous answer of "put sourcetype in input.conf". I already had that also (still not working).

0 Karma

Lucas_K
Motivator

That whitelist didn't help.

I put this into the app/local/props.conf.

[source::/path/file(.dat|.dat.gz)]
sourcetype = blah.

Then running ./splunk test sourcetype /path/file.dat.gz shows that it correctly sets the sourcetype to "blah". However the data still doesn't show up inside the index if the file is a .gz. I AM however able to get the .dat file in successfully if I manually uncompress it.

Trying a few other things as I really need to be able to process the logs using only splunk and not an external script to uncompress the files.

0 Karma
Take the 2021 Splunk Career Survey

Help us learn about how Splunk has
impacted your career by taking the 2021 Splunk Career Survey.

Earn $50 in Amazon cash!