This happens when you run Splunk's python without the Splunk environment. Please try $ /opt/splunk/bin/splunk cmd python ... View more

Typically, when a rotated logfile is compressed, Splunk will have already read the file in the uncompressed form, and will recognize the compressed form as data it has already dealt with. There are edge cases. If the start or end of the compressed file does not fully agree with the uncompressed version you might not get these neat and clean results, but they usually do. There are also potential problems if the file is deleted while splunk is reading it. (Compressing a rotated file means that the uncompressed version is deleted). Usually on Unix with local files this works fine, but on Windows or over some kinds of network filesystems splunk may get an error midway through the file and have to stop at that point. The full set scenarios for handling partially-read files that are then compressed is not handled, and in some cases you may see partial duplication of data. Best practice would be to keep at least one generation of the logfile, say logfile.1 in uncompressed form, to permit Splunk plenty of time to find and acquire the data from the uncompressed form. If you rotate to logfile.2.gz, Splunk should reliably recognize it as already handled data. If, however, you have extended downtime or other problems and unhandled data is stored in logfile.2.gz by the time things are sorted out, splunk will read and index the data from logfile.2.gz as well as logfile.1 and logfile. If you would rather err on the side of incomplete data rather than the possibility of some duplicate data, you could take the approach suggested by musskopf and simply not monitor the compressed files at all. ... View more

This isn't the answer you're looking for, so feel free not to accept. However, the general, deliberate, direction we're going in is having the searches not reload search state on every startup. This is because there's an large performance penalty which grows enormous in large environments to loading the state all the time. That said, yes there should be a developer-friendly poke-to-reload action. I'll try to ask around when work hours come by again. ... View more

Since you want to use the first part of the filename, you need to change your regex. REGEX=\w+(?=_\w+_\w+\.log$) You want the \w characters that precede the _number_number.log, so you have make them a capturing group, like so: REGEX=(\w+)(?=_\w+_\w+\.log$) Now $1 is ABCDE123 and similar. ... View more