Feature request... Is there any way to make a single extract that runs before KV_MODE = JSON kicks in? I keep seeing cases where just the body of the message is json, listed after the date, level and logger. In that case, you have to run the rex first followed by the spath. To the original question... If the whole body of the event is json, then the fields will automatically be extracted. It "just works." You can summarize like any other extracted field. I've never measured the performance, but it seems pretty good, but just think about what it's doing... you probably don't want to convert all of your logs to json. The only caveat I can give is to avoid complicated json documents if you can control the documents. For example, if you have arrays of objects that are actually each an event, you'll have to do some gymnastics with mvexpand to keep the fields in each nested event related. ... View more

It can't join the events because by the time it's parsed the event and sent it off to the indexer, there is nothing to tie them together anymore. That's the only way it can deal with massive data streams efficiently -- forget about what you no longer need to know. 🙂 I've actually been out of the Splunk admin game for a year or so. Maybe someone else will have a different idea... ... View more

The problem is "in streaming fashion". If you are writing events in chunks, say 4k chunks, every time the Splunk process hits EOF, it will call that the end of the event. When the next 4k chunk is written, the next event will start where it left off, somewhere in the middle of your event. I don't know a way around this problem, unfortunately. What Splunk really needs is a setting that says, "don't send the last event until you see this pattern at the beginning of a line." Or after a timeout. This wouldn't guaruntee anything, since your process could sit on the next chunk until after the timeout, but it would be better than the current behavior. You also mentioned copy and truncate. If you can get out of that business and simply use dated file names, you will also be better off. ... View more

On the search head, you will need this in fields.conf: [ActorsUserEmail] INDEXED=true And you DON'T want DEST_KEY = _meta in your transform. That said. Why do you think this needs to be an indexed field? There are really only a few cases where it is advantageous: 1. The thing you are extracting is in a metadata field (source, sourcetype, host) 2. The thing you need to search on is not an indexed term already, for instance a portion of a string. 3. The term you are looking for is very common, and creating the indexed field will drastically reduce the number of matched items. So, unless #3 is true, this indexed field isn't actually going to speed things up for you, and will really just make your index bigger. ... View more

This is old, so I'm sure you've figured it out, but a question deserves an answer... You need to use fillnull value="-" CVEID before the mvexpand to not lose records that have no CVEID value. ... View more

Here's python that would do it, but I was hoping there was already a command included that does that natively: import csv import sys output = [] csvreader = csv.reader(sys.stdin) for rowidx, row in enumerate(csvreader): for cellidx, cell in enumerate(row): if rowidx is 0: output.append([cell]) else: output[cellidx].append( cell ) csvwriter = csv.writer(sys.stdout) for row in output: csvwriter.writerow(row) ... View more

You could always do a test with Splunk itself. You could try V's app: http://splunk-base.splunk.com/apps/22339/field-perf-benchmark or just throw a bunch of data at it from some forwarders or even just on the disk itself. A query like this would give you a decent number: \* | eval bytes=length(_raw) | bucket span=1m _indextime | stats sum(bytes) as bytes_per_minute by _indextime | stats min(bytes_per_minute) avg(bytes_per_minute) max(bytes_per_minute) Run it over All time. ... View more

This just came up for me. Apparently the user has to have the "indexes_edit" capability. That's not so great. Indexes don't have permissions like other objects at this point. Perhaps they should? Read instead of adding read access at the role level? Write to allow collect to function, and therefore summary indexing? The confusing thing would be that this setting simply couldn't apply at index time, since events don't have permissions when they arrive at the indexers. ... View more

Instead of using the SideView Redirector module, use a couple of Search modules with SimpleResultsTable modules as children. Use a different layoutPanel so the tables drawn are where you want them. ... View more

If there are more than 20 megabytes left to read before EOF, then the forwarder will fall back to one stream, effectively. In most cases, this is fine and desirable, but in some cases it can be annoying. If neither the indexer nor your network are not the bottleneck, then I'm wondering if the default maxKBps is getting you. It is set quite low to prevent high CPU usage on forwarders. You can increase the value in limits.conf. The default is 256KB. [thruput] maxKBps = 256 Create a new app, called say YourCompanyForwarder, and make a limits.conf in local with a large number. YourCompanyForwarder/local/limits.conf [thruput] maxKBps = 256000 ... View more

There is a pid file mentioned in the script that should be modified in the second script, no? ... View more

So what's the best solution for many forwarders? "Stacked" Deployment Servers? Or can you put a load balancer in front of several other deployment servers? Will the checksums match if clients hit different deployment servers each time? ... View more

Something like this should do it: * | top limit=100 foo | eval a=1 | accum a | rangemap field=a 1-30=1-30 31-100=31-100 101-500=101-500 501-3000=501-3000 default=large | stats sum(count) by range ... View more

I think map may work, but it's certainly not efficient, and gives no indication of progress. In Jobs, you simply see "subsearch" for each search that it runs. |metadata type=hosts | fields host | map maxsearches=10000 search="search eventtype=foo host=$host$ | head 1" | fields _time host It may simply be more efficient to search for the eventtype over all time and use stats max(_time) by host. ... View more

Is this 20MB value tunable? I would like to have a forwarder reading from many files and fan them out to many indexers as fast as it can. The single threaded nature is killing me. ... View more

The scenario is this: 1. Splunk is tailing a file. 2. Splunk closes the file for whatever reason, whether it hasn't been modified for some time, or Splunk is down, or there are so many files that Splunk is having trouble keeping all the active files open. 3. Entries are written to the file while Splunk has the file closed. 4. Before Splunk reopens the file, the file is rolled and compressed immediately. The desire is for Splunk to handle this case, which unfortunately isn't that uncommon. ... View more

It thinks it has already indexed the entire file. INFO ArchiveProcessor - Archive with path="/tmp/foo/XCA_XCPD-2011-04-11T00_08_081e.gz" was already indexed as a non-archive, skipping. ... View more