Here is one hacked up way to do this. BTW, I'm going to let this unanswered for now, in the hopes that someone come up with a better (less-hackish) answer. Also, the ability to directly issue an eval/where expression from within a custom search command could be extended to many more situations than this limited hack. After some playing around, I came up with an approach that works by using a combination of getinfo and a dynamic preop command. This approach leverages the fact that the getinfo mechanism calls the script twice, and passes in the arguments to the script both times. The first __GETINFO__ call dynamically builds a pre-streaming operation based on the argument passed to the script. This causes the pre-operation command to execute an eval command before my custom search command is run; and so the temporary field is accessible from my search command when it's called in __EXECUTE__ mode. Below is a demo version of my script. If the "filter" eval expression evaluates to true, then function f1 is used to handle the event, and if it evaluates to false then function f2 is used instead. In this simple example, f1() adds two fields, and f2() multiples two fields. If no "filter" is given, then f2() is always called. The script is called like this: ... | mycmd filter="date_hour>12" | table a b sum product mycmd.py: import splunk.Intersplunk (isgetinfo, sys.argv) = splunk.Intersplunk.isGetInfo(sys.argv) args, kwargs = splunk.Intersplunk.getKeywordsAndOptions() # Process args eval_filter = kvargs.get("filter", None) if isgetinfo: # streaming, generating, retevs, reqsop, preop if eval_filter: splunk.Intersplunk.outputInfo(False, False, False, True, "addinfo | eval _filter=if(%s,1,0)" % eval_filter) else: splunk.Intersplunk.outputInfo(False, False, False, True, "addinfo") def f1(r): # Add field 'a' and 'b' r["sum"] = int(r["a"]) + int(r["b"]) return r def f2(r): # Multiply field 'a' and 'b' r["product"] = int(r["a"]) * int(r["b"]) return r if __name__ == '__main__': (results, dummyresults, settings) = splunk.Intersplunk.getOrganizedResults() if len(results) >= 1: search_id = results[0]["info_sid"] else: search_id = None logger.info("SID: %r", search_id) output = [] try: for result in results: if "_filter" in result: if result["_filter") == "1": result = f1(result) else: result = f2(result) del result["_filter"] else: result = f2(result) output.append(result) splunk.Intersplunk.outputResults(output) except Exception, e: splunk.Intersplunk.generateErrorResults("Unhandled exception: %s" % (e,)) ... View more

Here's the best approach I've come up with so far. I would still like a better way to do this, so I'll leave this question open for a while. Basically setup the search command to run the addinfo command first. I have supports_getinfo = True , so I used the following command in my script: # streaming, generating, retevs, reqsop, preop splunk.Intersplunk.outputInfo(False, False, False, True, "addinfo") Then it's simply a matter of grabbing the info_sid field from the first record, like this: (results, dummyresults, settings) = splunk.Intersplunk.getOrganizedResults() if len(results) > 0: search_id = results[0]["info_sid"] else: search_id = None Of course if there are no events/results found, then there is no first record. In my case I don't care because there's no work to do anyways. But I don't want an exception thrown just because there where no matching events in the base search. Update: It appears that in order for the preop command to be executed, you have to set retevs=False (that is "retainsevents"=0). I'm not sure why this is, and if this is an intentional behavior or a bug. Either way it's annoying since I want the event view, and not the results view. ... View more

If you look in alert_actions.conf , you'll see the argument searchid=$search_id$ passed to the sendemail search command. So this looks like $search_id$ seems to be inherently known within the alert actions environment. So that explains that example. But doesn't really help in my use case. ... View more

I'm not familiar with the "mon_*" prefixed fields, but then again I don't know all that much about how the si search commands summarized your fields either, so this could be normal. Well, at least you've been able to prove that it's not a summary indexing problem, it's something in your combination of sistats ... | stats ... , or it's a bug. ... View more

Sorry, but I think your missing the point. I know I can do whatever I want within python, but the point is that I don't want to go build by own eval-engine. I simply want to leverage the one Splunk already built. (The eval command already has over 40 functions and that list is growing each release. I certainly don't want to go build my own mini expression language and all those functions. And then try to keep that in sync with every new splunk release.) ... View more

Also, have you tried searching across all time? (Just in case there is some kind of timestamp recognition issue) ... View more

I setup our primary splunk partition as XFS simply because it allows for file defragging where as ext3 does not. Any kind of file fragmentation will impact performance because of extra seeks. With ext4 offering defrag support as well, perhaps that would be an option worth considering; although it's still a bit new, and apparently the defrag support isn't considered stable yet. Of course, splunk's file usage patterns seem to keep fragmentation fairly low. All the rawdata files can get fragmented, but then get gziped (and thus re-written to disk) which helps keep fragmentation low. Also the *.tsidx files can get fragmented as new content is added to a bucket, but then these files get merged over time, which again triggers the creation of a new file which is much less likely to be fragmented. You can look at some fragmentation of various files on your ext3 files system using the filefrag tool. You have to run this as root, and it only reports one file at a time, so I often run a command like: find /opt/splunk/var/lib/splunk -type f | xargs filefrag ... View more

Sounds like you are looking for the crawl command. You can use this as part of a scheduled search (run it daily to monthly, depending on how often new files could show up). The crawl command will scan your directory and look for potential log files. Something like this could do the trick: | crawl root="/var/log;/opt/log" | search isfile=True source="*/web.config" NOT status="added" | input add NOTE: You should take the " | input add " off the end until your confident that the search is going to do what you want. You may need to update crawl.conf , since it looks that bad_extensions_list includes "config" which would prevent this file from showing up. ... View more

Can you post your props.conf entry for [tomcat-catalina-out] as well? Also, any reason why you aren't monitoring the rotated files too? Sounds like you have a source normalizing transformer in there already, but the monitor stanza doesn't look at the rotated files. ... View more

I looked through system/default/props.conf and it appears that you simply have to have your source-based stanza point to a custom/bogus sourcetype, which is where you set invalid_cause = archive . I think an example may make more sense then the paragraph above. [source::....(tbz|tbz2)(.\d+)?] unarchive_cmd = _auto sourcetype = preprocess-bzip NO_BINARY_CHECK = true [source::....bz2?(.\d+)?] unarchive_cmd = bzip2 -cd - sourcetype = preprocess-bzip NO_BINARY_CHECK = true [preprocess-bzip] invalid_cause = archive is_valid = False LEARN_MODEL = false What I don't get is this: What's the need for all the different "preprocess-*" sourcetypes? I mean, why not just create a single [preprocess-archive] (or something like that) and then just point all the [source::...*] stuff to a single sourcetype. All of the preprocess-* sourcetype are identical in the system default file. I don't think you ever see these sourcetypes within splunk, do you? ... View more

For a simple count like that, I'm not sure that sistats buys you anything that plain stats wouldn't do. You do have to watch out for how you name your fields, but that's not a big deal. It would also be good to confirm that your scheduled interval and search time range aren't overlapping or anything like that. That could certainly lead to too many summary events as well. ... View more

By different types of files I'm referring to sourcetype, not different file extensions. If you want some files to be sourcetype=WEB and other to be sourcetype=EPP , then you have multiple "types" of files; then the approach above should scale well for you. ... View more

Are you going to be monitoring multiple files under /opt/log ? If so, then I would suggest a slightly different approach. Instead of hardcoding the sourcetype in your monitor stanza, you can use a props entry to establish your sourcetype based on file path patten. This way you can have a single monitor looking for a number of different types of files at once, and the correct sourcetype should be independently for your different types of files. Prior to 4.1, the approach I'm suggesting really was your only option. But in 4.1, you can now have nested monitor stanzas which removes the previous limitation; however, I still think there are advantages to this setup. props.conf: [source::.../epp_server/epp_server.log] sourcetype = EPP [EPP] # Whatever specific sourcetype settings that you need (TIME_FORMAT, SHOULD_LINEMERGE, ...) inputs.conf: [monitor:///opt/log] whitelist = (/log/\w+/epp_server/epp_server\.log|regex2|regex3)$ host_regex = ^/opt/log/([^\]+)/ Here you would replace "regex2" and "regex3" with patterns that would match whatever other paths you would like to monitor under this directory structure at the same time. If you have a very small number of files in this folder, you could easily just setup different source matching prop stanzas and not need a whitelist at all. If you only need to blacklist a few files, you can create a "blacklist" entry in your inputs.conf entry, or your can do this using props.conf path-based rule as well. The advantage of using the props-based approach is that it's automatically inherited by all of your inputs; you don't necessary have to update the "blacklist" on multiple monitor stanzas. The trick here is to use the special sourcetype called ignored_type which will prevent the matching file from being indexed. Here's and example that blocks files with the " \.old " extension: [source::.../epp_server/*.old] sourcetype = ignored_type ... View more

Note that your config will allow matching arbitrarily-nested directories. See southeringtonp's note about ... vs * . Also, and this is very minor, so FYI: you technically don't need to escape the forward slash (" / ") character. People often do because you see that frequently in a sed-like expression and and PERL regexes, but it's not actually necessary. That said, it doesn't hurt anything to have them there if you like extra back-slashes for whatever reason. 😉 ... View more

Keep in mind that monitoring like this will still scan many subdirectories under /opt/log , so if there are many files and directories under this directory even though only a few files will match the pattern. ... View more

BTW, It's perfectly acceptable to answer your own question, but you may have to wait a few days before you can "accept" your own answer. Which you should do. ... View more

Just wanted to point out that the question is less to do with Linux vs Windows, and more to do with BASH (or whatever shell you're using) vs the windows shell (cmd.exe). For example, on Linux you could to use a different shell and it could behave differently; but I would expect that most shell should work fine with single quotes or double quotes. On Windows you have less choices out of the box, but you can install and use bash (with cygwin, or something similar) if you wanted to. ... View more

There is a new search-time option for this scenario. In splunk 4.1.5 the tonumber() eval command was added. This function allows conversions between different bases which can convert from hexadecimal (base-16) to a standard (base 10) value. | eval dec_value = tonumber(hex_value, 16) Of course you'd have to do this many times depending on how many fields you have. ... View more