Getting Data In

How to monitor .dat.gz files?

Genti
Splunk Employee
Splunk Employee

I have set up an inputs.conf monitor stanza to index a few .dat.gz files but do not see them getting indexed.

There is no blacklist or null queue routing going on.
Also, there are no error messages in splunkd.log.

Tags (1)
0 Karma
1 Solution

Genti
Splunk Employee
Splunk Employee

There are two issues going on here.

  1. Why are the files not being moitored and how to actually monitor them
  2. Why are there no errors in splunkd.log

First, Splunk will not index binary files. Even if you have a file.dat.gz that is not binary, and you specify NO_BINARY_CHECK, Splunk will still fail to index the file.

This is because there is stanza in /etc/system/default/props.conf that tells splunk not to.

[source::....(0t|a|ali|asa|au|bmp|cg|cgi|class|d|dat|deb|del|dot|dvi|dylib|elc|eps|exe|ftn|gif|hlp|hqx|hs|icns|ico|inc|iso|jame|jin|jpeg|jpg|kml|la|lhs|lib|lo|lock|mcp|mid|mp3|mpg|msf|nib|o|obj|odt|ogg|ook|opt|os|pal|pbm|pdf|pem|pgm|plo|png|po|pod|pp|ppd|ppm|ppt|prc|ps|psd|psym|pyc|pyd|rast|rb|rde|rdf|rdr|rgb|ro|rpm|rsrc|so|ss|stg|strings|tdt|tif|tiff|tk|uue|vhd|xbm|xlb|xls|xlw)]
sourcetype = known_binary

Note that this is not just happening for .dat files but happens for all the above extensions..
In order to actually have the files indexed then we need to create (or edit) a props.conf within /etc/system/local/ with the following stanza:

[source::....(dat)]
sourcetype = mysourcetype
priority = 20

This will overwrite the default/props.conf and will tell splunk to use a different sourcetype for the .dat.gz files, and hence allow them to actually be indexed.

The reason why there was no message in splunkd.log is because of a bug, which has been filed and our friendly devs will soon be working on it.
Hope this helps someone out there..
.gz

View solution in original post

Genti
Splunk Employee
Splunk Employee

There are two issues going on here.

  1. Why are the files not being moitored and how to actually monitor them
  2. Why are there no errors in splunkd.log

First, Splunk will not index binary files. Even if you have a file.dat.gz that is not binary, and you specify NO_BINARY_CHECK, Splunk will still fail to index the file.

This is because there is stanza in /etc/system/default/props.conf that tells splunk not to.

[source::....(0t|a|ali|asa|au|bmp|cg|cgi|class|d|dat|deb|del|dot|dvi|dylib|elc|eps|exe|ftn|gif|hlp|hqx|hs|icns|ico|inc|iso|jame|jin|jpeg|jpg|kml|la|lhs|lib|lo|lock|mcp|mid|mp3|mpg|msf|nib|o|obj|odt|ogg|ook|opt|os|pal|pbm|pdf|pem|pgm|plo|png|po|pod|pp|ppd|ppm|ppt|prc|ps|psd|psym|pyc|pyd|rast|rb|rde|rdf|rdr|rgb|ro|rpm|rsrc|so|ss|stg|strings|tdt|tif|tiff|tk|uue|vhd|xbm|xlb|xls|xlw)]
sourcetype = known_binary

Note that this is not just happening for .dat files but happens for all the above extensions..
In order to actually have the files indexed then we need to create (or edit) a props.conf within /etc/system/local/ with the following stanza:

[source::....(dat)]
sourcetype = mysourcetype
priority = 20

This will overwrite the default/props.conf and will tell splunk to use a different sourcetype for the .dat.gz files, and hence allow them to actually be indexed.

The reason why there was no message in splunkd.log is because of a bug, which has been filed and our friendly devs will soon be working on it.
Hope this helps someone out there..
.gz

rakesh_498115
Motivator

Hi ..i am facing the similar issue for indexing dat files..i have removed the dat extension in system/default/props.conf file ..but then its not indexing...pls help

0 Karma
Get Updates on the Splunk Community!

Why You Can't Miss .conf25: Unleashing the Power of Agentic AI with Splunk & Cisco

The Defining Technology Movement of Our Lifetime The advent of agentic AI is arguably the defining technology ...

Deep Dive into Federated Analytics: Unlocking the Full Power of Your Security Data

In today’s complex digital landscape, security teams face increasing pressure to protect sprawling data across ...

Your summer travels continue with new course releases

Summer in the Northern hemisphere is in full swing, and is often a time to travel and explore. If your summer ...