How to monitor .dat.gz files?

[Genti]

I have set up an inputs.conf monitor stanza to index a few .dat.gz files but do not see them getting indexed.

There is no blacklist or null queue routing going on.
Also, there are no error messages in splunkd.log.

[Genti · Accepted solution]

There are two issues going on here.

1. Why are the files not being moitored and how to actually monitor them
2. Why are there no errors in splunkd.log

First, Splunk will not index binary files. Even if you have a file.dat.gz that is not binary, and you specify NO_BINARY_CHECK, Splunk will still fail to index the file.

This is because there is stanza in /etc/system/default/props.conf that tells splunk not to.

[source::....(0t|a|ali|asa|au|bmp|cg|cgi|class|d|dat|deb|del|dot|dvi|dylib|elc|eps|exe|ftn|gif|hlp|hqx|hs|icns|ico|inc|iso|jame|jin|jpeg|jpg|kml|la|lhs|lib|lo|lock|mcp|mid|mp3|mpg|msf|nib|o|obj|odt|ogg|ook|opt|os|pal|pbm|pdf|pem|pgm|plo|png|po|pod|pp|ppd|ppm|ppt|prc|ps|psd|psym|pyc|pyd|rast|rb|rde|rdf|rdr|rgb|ro|rpm|rsrc|so|ss|stg|strings|tdt|tif|tiff|tk|uue|vhd|xbm|xlb|xls|xlw)]
sourcetype = known_binary

Note that this is not just happening for .dat files but happens for all the above extensions..
In order to actually have the files indexed then we need to create (or edit) a props.conf within /etc/system/local/ with the following stanza:

[source::....(dat)]
sourcetype = mysourcetype
priority = 20

This will overwrite the default/props.conf and will tell splunk to use a different sourcetype for the .dat.gz files, and hence allow them to actually be indexed.

The reason why there was no message in splunkd.log is because of a bug, which has been filed and our friendly devs will soon be working on it.
Hope this helps someone out there..
.gz

[rakesh_498115]

Hi ..i am facing the similar issue for indexing dat files..i have removed the dat extension in system/default/props.conf file ..but then its not indexing...pls help