Thanks gekoner. Yeah this is a weird one. I/O hopefully isn't the cause as its a dedicated filer for this purpose, and we don't have any users on the new cluster yet (or data coming in). Interestingly, before I had configured mounted bundles, I had added these indexers as normal search peers, and all worked properly. So definitely this is something to do with Shared Bundles, the filer, directory paths/permissions, etc. I wish the docs were more explicit in this area. I've opened a support ticket with splunk, I will post an answer here after we've worked it out. ... View more

I would have just run the backfill script on the search head. Aslong as it is outputting events to all of those indexes anyway the results will be distributed to all the available indexes as per normal. The limitation that I have found is that the max number of cpu's you can use for this is 16. I believe this is a hardcoded limit in the fill_summary_index.py script itself. I am unsure of the reason for this and if you could just change the number in line 268. The F5 vip is just the web front end your balancing right? If so then it has no bearing on backfill data. Your search heads should be configured for distributed search anyhow. This way all the search heads will know which search head is performing what search so everything should be in sync. As far as I have seen the backfill script won't populate jobs to be distributed to other search heads other than the one your running the script on. So to get the best performance from all your available sh'es you might have to do some intelligent division of jobs across the available search heads's. Easiest would be by time and different savedsearches. replication on the other hand ... i'll leave for someone else to answer 😉 (v5.0.1 based summary indexes?) ... View more

Hey guys I have a follow up question similar to this ask. I'm familiar with how transaction work, but i'm having fits trying to find or calculate the difference between duration events. Transaction groups "like" events, then sums the time within that event grouping. Ok, enough of a rehash of the manual. My question is this. has someone come up with a way to find the duration "between" the durations? Here is my result set i'm working with. The ask is how do I get a running total of the difference 1 and 2, then 3 and 4. Of course i'll want to display this in my chart as well:D Here's what I have so far - eio | rex "(?i)] [(?P [^\ ]+)\s+] [(?P [^\ ]+)" | transaction stateio startswith="issuedio=PauseIO" endswith="issuedio=ResumeIO" | eval durationstr=tostring(duration,"duration")| rename durationstr as "PauseIO Durations" | table"PauseIO Durations", _time 1 [requested ] [ResumeIO Live] [completed ] [ResumeIO Live] duration=18 2 [requested ] [PauseIO Live] [completed ] [PauseIO Live] duration=17 3 [requested ] [ResumeIO Live] [completed ] [ResumeIO Live] duration=18 4 [requested ] [PauseIO Live] [completed ] [PauseIO Live] duration=17 Thanks in advance. ... View more

Yep i'm with ya dude. And I see you've employed my good friend relative_time. I too am not a big fan of append or appendcols. I've run into major problems sorting "appended" results so I've shied away from it's use. Hence relative_time. I will certainly muck with this. I only chose the gauge function because i'm looking for pretty graphics for manager types;-)... Thanks for the reminder about the coolness of the if-relative time combo... ... View more

Ah, fixed that in mine. Thanks for pointing it out 🙂 ... View more

I'd like to piggy back off of this post. I apologize if this is bad netiquette. Can someone suggest how to use stats rather than append for this search? I stole the structure from the deployment app index=_internal source=*license_usage.log pool="auto_generated_pool_enterprise" earliest=@d| eval GB=b/1024/1024/1024 | eventstats sum(GB) by pool | timechart partial=f span=30m per_hour(GB) as GBph | eval marker = "today" | eval _time = _time+1800 | append maxtime=600 maxout=1000000000 timeout=600 [ search index=_internal source=*license_usage.log pool="auto_generated_pool_enterprise" earliest=-1d@d-30m latest=@d-30m| eval GB=b/1024/1024/1024 | eventstats sum(GB) by pool | timechart partial=f span=30m per_hour(GB) as GBph | eval marker = "Yesterday" | eval _time = _time+86400*1+1800 ] | timechart median(GBph) by marker Sorry I had to use an answer post because the comment text box is too limiting. Thanks in advance, Curtis ... View more

Hey Lowell my journey with the transaction command and my above dilemma has been a fun bit of investigation. eval and transaction are my friend. No longer to be feared. Thanks for your comments and suggestions. Thanks again ... View more