https://community.splunk.com/t5/Getting-Data-In/Log-rotation-for-compressed-files/m-p/76944#M15731 <P>You can add whitelists/blacklists to your inputs.conf to filter out unwanted files:</P> <PRE><CODE>blacklist = \.(gz)$ </CODE></PRE> <P>Should filter out anything in the folder with a .gz extension. (Or you could just whitelist .log files to get the same result. Depends on what else is in there)</P> <P><A href="http://docs.splunk.com/Documentation/Splunk/4.3.2/Data/Whitelistorblacklistspecificincomingdata">http://docs.splunk.com/Documentation/Splunk/4.3.2/Data/Whitelistorblacklistspecificincomingdata</A></P> Fri, 15 Jun 2012 20:45:07 GMT emiller42 2012-06-15T20:45:07Z https://community.splunk.com/t5/Getting-Data-In/Log-rotation-for-compressed-files/m-p/76943#M15730 <P>We are using Splunk to monitor server.log file from a JBoss instance that rolls over daily (we use the logrotate utility to gz server.log daily)</P> <P>The folder looks like this inside : </P> <PRE><CODE>//var/log //server.log //server.log.June-12.gz //server.log.June-13.gz //server.log.June-14.gz //server.log.June-15.gz </CODE></PRE> <P>//</P> <P>We use the universal forwarder on this linux box to push data out to the indexer.</P> <P>Currently: Our configuration in the inputs.conf on the forwarder side looks like this. </P> <PRE><CODE>[monitor://var/log/jboss_logs/server*] disabled=0 index=os sourcetype=serverlog </CODE></PRE> <P>What this does unfortunately is that <STRONG>it gets the daily server.log (which its supposed to because of the server* wildcard) -- and then, everyday it indexes the uncompressed content of the server.*.gz files that are out there</STRONG></P> <P>Based on what is described here - apparently log rotation does not apply to the .gz and .tar file formats because they are treated as new files:</P> <P><A href="http://docs.splunk.com/Documentation/Splunk/latest/Data/MonitorFilesAndDirectories">http://docs.splunk.com/Documentation/Splunk/latest/Data/MonitorFilesAndDirectories</A></P> <P>Does this mean that we will definitely see duplicates ? Has anybody seen a problem like this previously ?</P> Fri, 15 Jun 2012 18:53:27 GMT https://community.splunk.com/t5/Getting-Data-In/Log-rotation-for-compressed-files/m-p/76943#M15730 asarolkar 2012-06-15T18:53:27Z https://community.splunk.com/t5/Getting-Data-In/Log-rotation-for-compressed-files/m-p/76944#M15731 <P>You can add whitelists/blacklists to your inputs.conf to filter out unwanted files:</P> <PRE><CODE>blacklist = \.(gz)$ </CODE></PRE> <P>Should filter out anything in the folder with a .gz extension. (Or you could just whitelist .log files to get the same result. Depends on what else is in there)</P> <P><A href="http://docs.splunk.com/Documentation/Splunk/4.3.2/Data/Whitelistorblacklistspecificincomingdata">http://docs.splunk.com/Documentation/Splunk/4.3.2/Data/Whitelistorblacklistspecificincomingdata</A></P> Fri, 15 Jun 2012 20:45:07 GMT https://community.splunk.com/t5/Getting-Data-In/Log-rotation-for-compressed-files/m-p/76944#M15731 emiller42 2012-06-15T20:45:07Z