You might also find eventstats useful here, if your looking for some data trending over a few events. ... View more

I'm guessing that WAS_installation_path , WAS_profile_name and WAS_cell_name are variables that Nicholas is trying to have replaced. Nicholas, care to jump in here? ... View more

Keep in mind that this is a new feature in Splunk 4.1. ... View more

Anyone inside splunk know if High CPU utilization with LFW and deployment client (SPL-26789) is still open? The workaround at the time (4.0.6ish) was to disable the deployment client..... ;-( ... View more

I think there is an issue with the Light Forwarders not disabling the web interface, as you point out. I think you need to set the service to disabled (or manual) by hand.... Not sure if this is scheduled to be fixed or not... Someone please correct me if I'm wrong. ... View more

Hey Emma, I'm seeing it in the docs now. (I don't know if docs were updated, or I was just missing it before...) I know I've seen it mentioned on other questions on this site, so other people have run into this as well. What confused me in the known issues is that the message doesn't indicate that the maximum has changed between 4.0 and 4.1. I think that would be helpful to point out. ... View more

If you are going to change maxHotSpanSecs , it may be worth not jumping down to 1 day all at once. It may be beneficial to try, for example, a 1 week window first. Then you can lower it more if you like the results after a few weeks. However, I suspect that you may be surprised at how often events come in for an older date range, in which cause you could end up creating tons of buckets.... of course this is highly specific to your environment, which I why I'm throwing out the take-it-slow approach... An alternative to this is having your scheduled backup script kick of an explicit bucket roll (e.g. $SPLUNK_HOME/bin/splunk search '| debug cmd=roll index=index_name' ) before doing the backup. This is an approach that we used to use, but now we rely on LVM snapshots, which has been working pretty well. Another consideration is that a bucket roll in Splunk 4 means that ALL hot buckets are rolled, so this has the potential to create even more buckets... If you haven't look at the backup topic on the Community wiki, I reccomend this page: http://www.splunk.com/wiki/Community:BestPracticesForBackingUp ... View more

Nick, if you know of a better solution please post it as an answer. I'd love a better way to do this. But, from everything I've seen, Splunk (up through 4.1x) does not allow field extractions where the field value is composed of multiple regex extraction groups. Yes, I fully agree that using a second search will be slower, but it doesn't work without it. (Which is why I refereed to it as a "workaround", which I feel is the most accurate description. (This does work when using an indexed field, but using indexed fields are generally discouraged.) ... View more

You should probably avoid using dashes, since I don't think they are legal. And even if they are, that would make eval's ugly, for example what does eval total_kb=websphere-bytes/1024 mean? ... View more

First off. It should be pointed out that splunk resuming indexing where it left of is a highly desirable feature. This is what lets splunk keep going after a splunk restart, for example. Here is another possible solution. It may not work. Setup all your inputs with followTail=1 Right before the testing begins. Shutdown splunk. Clear all event data from the fishbucket index. When the tests are done. Start up splunk again. Now when splunk starts, it should read all of your log files from the end and not from where it left off when it was shutdown. The biggest problem I see with this approach, is that you can't selectively clear the fishbucket database. So this would take effect for ALL files. If there were instead some way to trick splunk into thinking that it had already read to the end of a file, that would be much more beneficial. Unfortunately, I don't know of a way of doing that. (If you knew the right kind of events to feed into into the fishbucket, this may be possible... but this is way over my head.) Another option. Forward all your logs via syslog to splunk. Then shutdown splunk during the tests, and start it back up afterwards. Since syslog forwarding will drop any event while splunk is down, this could work pretty easily. Additional, you could just setup a local firewall to b block and unblock the syslog traffic instead of actually bringing splunk up and down; then splunk would still be up for other activities... This will not work with a splunkforwarder though, because it buffers the log events so you don't loose them. Which again, is almost always what you want splunk to do. Yeah, you're asking for something splunk tries really hard not to do, drop events. ... View more

If your test are on a regularly scheduled interval. You could send your events to the nullQueue using a regular expression that only matches for certain hours, or better yet if there is something in the event's themselves to indicate that it's test data, then you could matching on that. ... View more

Good point about time inversed index stuff. My point is not to debate splunk vs relational DBs, just to note that there are options. (Sometimes people use a screwdriver to hammer in nails and then they somehow think it's the screwdriver's fault when it's not optimal... not that this is a perfect analogy. But we certainly don't want people throwing out their screwdrivers because of an initial bad experience.) ... View more

So, any idea on why this is done, what advantage it provides? ... View more

I was hoping for something more integrated than this. But this is fine starting point. Thanks. ... View more

Yeah, the version number and build in a single log message at startup time would be helpful too. 😉 ... View more

Here are two approaches, not sure which will work best for you: Search 1: your search | transaction fields="field1,field2,field3" | search duration>0 Search 2: your search | stats dc(_time) as times, values(_time) as time_values, by field1, field2, field3 | search times>1 | convert ctime(time_values) It's not clear from your question if dateA and dateB represent two different dates or two different field names. In the searches above, I'm assuming that your actually talking about the timestamp of your event, which is accessible in the _time field. If you do have two field names, then perhaps this could work for you: your search | transaction fields="field1,field2,field3" | where dateA!=dateB Note: You have to use where when you are comparing the value of two different fields. You can use search if you are comparing against a constant value. ... View more

Let me add a few thoughts to what Jrodman said. I think the first and most important question is whether or not your field list will be completely static or if it will change. If it changes then the k=v option is a better and more flexible approach. If you have a few variations, it's best to make one sourcetype per set of fields. Also you should keep in mind that with the k=v approach, you can always do a direct search against "v3=c" (which will be an indexed term in the kv, but it will NOT be in the CSV approach.) If you're new to splunk, I should point out that searching for "v3=c" (with quotes) is in fact different from searching for v3=c1 (without quotes). The first is an exact string matching type of search. And with normal segmentation rules, it will be a faster lookup if the term c1 commonly occurs in your data other than in the v3 field. The second example is a field search, which requires field extraction. Again, it all depends on the frequency of terms and how you need to search on your data. This doesn't matter so much for pure extraction speed. So if finding specific records is your goal, then there are some advantagse to the kv approach. But if your actually searching requires are more date based, for example, and your looking to do statistical analysis, then either approach will probably be about as fast and the CSV approach may be more applicable for you. (Jrodman, do you know if FIELDS based extraction is faster or slower than the automatic kv extraction. It seems like CSV could be faster from a parsing perspective, but a more definitive answer on this would be great) Bottom line. If this is really such a big deal. Then I would recommend testing out both methods. Create two temporary indexes, feed the same data into both (obviously formatted in your two different ways). Also be sure to make a different sourcetype for each test. Make sure that in the CSV sourcetype you set KV_MODE=none in props.conf and rely only on using the FIELDS in transforms.conf (you don't want the overhead of extracting KV pairs from your CSV formatted sourcetype). Then compare the speeds. This shouldn't take too long to setup. Then pick the fastest most ideal solution for you. (Note that comparing bucket sizes is tricky. You would want to rotate your buckets, manually run splunk-optimize , and then manually gzip all of the files in the rawdata folder that don't end with `.gz' already. ... and there could be more involved in a truly fair comparison.) For a speed test, you may also want to try uncompressing all your .gz files in the rawdata folder. I've heard this can sometime improve performance, but it could also backfire. You'd have to try to know for sure. Also, if your data is really static (well defined, and strongly typed), then you may be better off feeding your data into a database rather than splunk. Don't get me wrong, I love splunk, it does some really awesome stuff, but I strongly believe in using the right tool for the job. (Hope I don't get in trouble for saying this.) You can probably find a way to pull data back out of your database and feed it back into splunk if you want to leverage splunk's visualization features. It's certainly feasible to to write a small (data generating) search script to pull data from your database and use splunk to post-process or visualize it. Just a thought. There are lots of pros/cons for either approach, but only you know the whole picture. ... View more

Just to follow up on jrodmans comment about stale sentinel files. Perhaps some one will find this useful. Here is a short script I wrote which searches your index folder for .sentinel files without a matching bucket. You can use the output to quickly remove these files (or event have them be removed directly by uncommenting a line in the script). Code listing for lone_sentinel.py : #!/usr/bin/python import os SENTINEL = ".sentinel" def find_sentinels(path): for (root, dirs, files) in os.walk(path): if "rawdata" in dirs: dirs.remove("rawdata") for fn in files: if fn.endswith(SENTINEL): yield os.path.join(root, fn) def find_lone_sentinels(path): for sentinel in find_sentinels(path): bucket = sentinel[:-len(SENTINEL)] if not os.path.isdir(bucket): print sentinel # To remove these files, uncomment the next line #os.unlink(sentinel) if __name__ == '__main__': import sys if len(sys.argv) > 2: path = sys.arvg[1] else: path = os.environ["SPLUNK_DB"] find_lone_sentinels(path) ... View more

That's a great question Chris. I'm looking forward to a great answer too; I'd like to understand the fishbucket index better as well. ... View more