A lookup script is not appropriate because it runs at the time of search. If the underlying data is changing and you need the state from some time in the past, then you can't use it in that way, regardless of performance. What you essentially need is some way of capturing the state of the DB at the time of the logging (or shortly after that time). The basic problem is, you want information that you are not recording but instead just throwing away. (I don't even know why that database exists. Is it for real that it's that badly designed, or is it that you're simplifying and making it seem so?) The only solution to this is to record that information. The "right" way to do this in my mind is to put the information into the order information log in the first place. I suppose you could do this with a scripted input. We can try to come up with other solutions, but in the end they all amount to getting and storing that mapping, and the only difference is where it is stored. In my "right" way, it is stored directly in the order record. If it's impossible to make the changes to the source log, and Splunk is our only tool to make this work, we could also try to record every single change to a row in the database, and then correlate back by time and order id at the time of search. Or we could have a scripted input that watched incoming Splunk logs of the orders and more selectively did lookups, possibly in batches. This amounts to using Splunk to completely replace that database, both by actually storing the data and by not being slow. If the orders come in at a high rate, and the DB takes a half second to respond to each query, any of these solutions is going to cause problems when we try to capture DB state at index time. So in the course of trying to use Splunk to replace the (incomplete and slow) database we will be trapped in the bottleneck of having to be getting every state change out of said slow database. I don't consider this a "wrong" solution because it's just overengineering a complex and fragil workaround to a problem that should be fixed at its source. Fundamentally, this application is broken around this database, and Splunk is not really the right solution. ... View more

props.conf TIME_PREFIX = \<CreationDate\> TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N SHOULD_LINEMERGE = false LINE_BREAKER = \>\s*(?=\<row\>) REPORT-xmlext = xml-extr transforms.conf [xml-extr] REGEX = \<(\w+)\>([^\>]*)\<\1\> FORMAT = $1::$2 MV_ADD = true REPEAT_MATCH = true should do it. ... View more

If it's a one time re-import, I recommend you use first delete the old source with the search command: source=/full/path/to/file | delete (remembering that the "delete" capability is not granted to any role including admin by default) and then use the oneshot input at the command line, or it's equivalent in the Manager>Data Input>Files and Directories,option "Index a file on the Splunk server". The cli is: ./splunk add oneshot /full/path/to/file -sourcetype mysourcetype -index myindex -host myhostparam The parameters other than the full path to the file (not a relative path) are optional. This is preferable to setting crcSalt to <SOURCE> for a one-time reload, since you don't need to then set it back if you don't want that enabled. ... View more

Yes. Well, actually the default index used is the default db which is almost always main . Note BTW that | dbinspect does not distribute, i.e., it only reports on the local server even in distributed search mode. ... View more

The biggest thing you can do is reduce the size of the internal built-in indexes, particularly the _internal db. You can also edit log.cfg to reduce the number and size of retained text log files. splunkd.log, splunkd_access.log, metrics.log are allowed up to 150 mb each, for example. I suppose you could delete all the *.pyc and *.pyo files under the Python2.6 directory. They would be regenerated as needed, but without SplunkWeb running, there might not be as many of them needed, and there might be some extraneous ones anyway. I'd say to use find . -name *.pyo -o -name *.pyc | xargs rm -f but Windows. Looks like that will save you, oh, maybe 4 or 8 MB. Pretty much nothing else large that you can afford to delete. ... View more

Search time configurations, including lookup tables, lookup scripts, and custom search commands, as well as field extractions, tags, event types, aliases, etc. go on the search head and the search head only. The Distributed Search mechanism will make sure the configuration items are sent to the indexers when a search is issued. However, do note that lookup tables and lookup/search scripts must be in an app or system lookup or bin directories. Also any resources that scripts themselves may reference will only be copied to the indexers if they are files located inside of the bin or lookup other app folders, and that such references must be relative to the app or script base (and not absolute). (Other resources will only be available if you use some other method to get them the indexers and reference them accordingly in your scripts.) If you have scripts, you may rely on this mechanism to distribute the scripts, or you can look at the localop command and local option on the lookup search command. ... View more

The time is extracted where the log data is parsed. This is on the indexer if you are using a lightweight forwarder, and on a forwarder if you are using a heavy forwarder. (Parsing the data is the essential difference between a light and heavy forwarding.) Update: I wrote this which goes into a more thorough explanation. ... View more

For a specific search, you can use the head search command in your search. This does not limit users or search results globally. There is no limitation on numbers of results in version 4.0. However, you can limit users by configuring a search window time limit for the user's role. This can be done in the Splunk Manager>Role, or using the srchTimeWin setting in authorize.conf. ... View more

Though you don't actually say, I assume your problem is the events with EventCode 540 are not being dropped and that you want them to be. I do not know of any bugs in this area. However, if that is what you are trying to do, one problem is that the " EventCode " you are presumably looking for is usually found at the beginning of a line, while the regex in your configuration requires a " . " before it. By default, " . " does not match line breaks, so your regex will not match what you intend. A regex that does do what you probably want is " (?m)^EventCode=540 ". ... View more