Let's get started by me asking this question.  If you were to do this by hand, using those two records above, how would you do that?  If CLIENT_IP_ADDRESS is actually that word, I don't see how they even connect.  You can't do it by hand, and Splunk can't really do it either - there's no magic involved. If CLIENT_IP_ADDRESS is a stand in, and actually IS 192.168.10.10, then we have a chance. You'll probably need a better regex than this (regex101 is your friend!) because I'm making it up anchoring on something in the middle that might or might not be a good anchor, and I'm too lazy to build a regex from the beginning because I'm not actually sure that's a 100% accurate sample event.  But here it is. (search that returns the first set of events) OR (search that returns the second set of events) | rex "Info: \d+\.\d+\s+\d+\s+(?<client_ip>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})" | eval client_ip = SrcIP | stats list(DstIP) as Destination_IP_addresses BY client_ip There's two searches in the first line, connected together with an OR - you have to pile all the data into one place, then let Splunk sift through it and connect things together as required. The second line uses a regular expression to pull a new field called "client_ip" from what *might* be the IP address in there.  This will need fiddling with possibly, as mentioned above. The third relies on the fact that there is no "client_ip" in existence in any event with a field named SrcIP (an assumption, but seems legit given what I can see?), so you just create a new client_ip from SrcIP.  Between line 2 and line 3, we've now let both "sides" have a client_ip that matches each other, so we can group on them in a variety of ways. Line 4 uses stats, which probably should be called "group" in this case because really that's what we're doing, to list all the DstIP addresses by client_ip. For that last line, you can use values(), list(), or any other stats calculation, on any number of fields all at once.  Like "values(DstIP), sum(initiatorBytes), list(Protocol)..." Look up "splunk stats site:docs.splunk.com" and it should totally give you all sorts of good advice. The same mechanisms can be used to include other fields in the output, or to join things up in slightly different ways.  Hopefully this gets you your answer, but even if not I hope it does get you started on fine tuning your question for us, we're happy to take another pass at it! Happy Splunking, Rich ... View more

... | bin span=1s _time | stats count by _time | stats max(count) as "winner winner chicken dinner" That's probably your basic need right there.  Change "span=..." to whatever you need.  'count' is probably what you want to use for the counts, but the peak rates - you don't tell us how those even might be calculated. It might be "max(fieldname)" or something else.  Note, I think by second probably isn't gonna be a very fast thing to do over a longer period of time.  There are 86,400 if those it'll calculate every day... Report acceleration may help, and/or building a data model and accelerating that.  Or using tstats if they're indexed fields you are fiddling with.  Lastly, this might actually be a good use for using a summary index. All of those things are easily able to be found with a search of (using google syntax, but I think most search engines follow that?) 'Splunk report acceleration site:docs.splunk.com' (obviously, change the keywords in there). Happy Splunking, Rich ... View more

I think if I am reading this right, you are using `tag!="snooze"` when you actually want `NOT tag="snooze"`. The former, `tag!="snooze"` returns all events where there is a tag defined and exists, and where that tag is not "snooze." The latter, `NOT tag="snooze"`, returns all events that do not have a tag that isn't equal to "snooze", or in fact that may not have a tag defined at all.  Doesn't matter - no tag defined means it's certainly not defined as "snooze" right? So use `NOT tag="snooze"` and see if that works better for you. https://docs.splunk.com/Documentation/Splunk/8.1.0/Search/NOTexpressions Happy Splunking! -Rich     ... View more

I'm not sure that question is quite clear enough to answer.  It feels like you started this explanation in the middle, then forgot to finish it. I get a glimmer that maybe you want a way to perhaps have your time segments chopped up into a maximum of 3 minute chunks to make things reset back to normal regularly, but I don't know to what you are referring to when you say "green" (and it's probably not important) Your output doesn't seem to be directly related to the two events you included?  There's some minor match on a bit of them, but not enough to see what it is you want to have happen. This is OK, though - we just need a bit more information! So what I recommend: Think through this problem from our point of view.  What is it that *we* need to know in order to answer this question? Remember, we don't know your environment, the dashboards you are looking at (I think that's where "green" comes in) or what this data is like except what you tell us of it.  But most of that probably isn't important anyway except the data itself, and the desired output! Did you provide a sample set of data that's "big enough" to illustrate the situations you need to cover (I'd guess that would take 5 to 20 events - please use the code button to paste them in so they're more compact and we can isolate them easily)? Does your expected output match (mostly, anyway) the sample data you presented?  Is it clear *how* it's related, and how we come to the output (not SPL-wise, but I mean, did you explain it clearly?) Do you have a start at a search, what have you tried already, what wasn't right about it? I have found that asking the question well takes time - many minutes (an hour isn't unusual!) of reviewing what it is you have, what it is you are after, and what you've tried so far.  In the process, often you solve your own problem by just doing this and breaking it down into all the little pieces and putting them back together.   But when you can't answer it with this process, you then have all the information to ask a precise, well thought out question that we can answer quickly and accurately.   ... View more

proof: Also note, in THAT case the prediction and the old Volume line up.  But normally they're not that lined-up-ed, those predictions are all over at that point.  🙂 ... View more

My first thought was their timestamps are being handled incorrectly, or they're wrong in the data.  This is sometimes because they're just plain wrong (Wouldn't be the first time I've seen them wrong *in the events*, but usually either that you are using the wrong field for _time, or because of timezones are off or not being recognized. Take a concrete example -  An event comes in now, but the timestamp says now+1 hour. _indextime is set to 1605131264  and _time is set to 1605131264 + 3600 = 1605134864. For most searches, this won't show up until an hour from now. And at that time, the indexing "lag" will be 1605131264 - 1605134864, which is -3600, ... So your search to find max(_indextime) will not probably ever point that one out.  0 is larger than -3600. There are two ways to find these: You could just do an abs() on the eval part, like this, to remove that negative sign. ... | eval indextime=strftime(_indextime,"%Y-%m-%d %H:%M:%S") | eval delay=abs(_indextime-_time) | timechart span=1m max(delay) But it might be useful to see if this is the case directly, in which case you can leave off the abs() and instead just timechart both the max and the min of the delay like in the below: ... | eval indextime=strftime(_indextime,"%Y-%m-%d %H:%M:%S") | eval delay=_indextime-_time | timechart span=1m max(delay) min(delay) Then you can see both sides of that chart and see if this is indeed what's happening. If this is what it is, then resolving it nearly always involves revisiting your timestamps and when you find what may be smoking guns in there, it might be best to create a new question to handle that topic. Happy Splunking! -Rich   ... View more

It sounds like you have consulted the documentation, so the next step is to file a case online at http://www.splunk.com/page/submit_issue They'll be able to give this the attention it needs probably better than we could.  Please post back what they found as an answer here, so that others may learn from it too! ... View more

Frankly, this sounds like a great place to use Splunk Cloud.  Fling all your data at Splunk Cloud from wherever you collect it.  Throw a UF on each log collection node, point its output to your Cloud instance.  Problem solved. But the same can be done without cloud too.   Mostly the same as above with a UF on each item that accumulates logs that you want to read, only this time point it to your indexing tier.  Your indexers must exist somewhere, so you'll need to make exceptions in your firewall for that location for the incoming 9997 traffic from the UFs from the IPs they're expecting to send from, and set up certificates and all that fancy stuff.  And please don't use an intermediate forwarder to aggregate it into a single connection - just open the "holes" (I hate calling them that) to all indexers and let Splunk do the work to LB between the indexers- it will work better, faster, and be no less secure. Then your one central location (which can be inside P1 if that's where it lives now) gets all the logs. ... View more

Not directly, but... Could you set that API to use a user/role that's specific to it and set that role's concurrent search limit to 1? ... View more

Ask the person who requested those to be ingested. Event IDs can be duplicated for different purposes across many different event logs, so a System 3039 may exist, and a Security 3039 may exist, and they may be completely different types of events.  You absolutely have to know which event 3039 they want you to ingest. Happy Splunking -Rich ... View more

Yes, maybe?  Probably, in fact. But I'm struggling to figure out what you are even doing.  It's like you are smashing several csv files together, sideways, doing some filtering on each then turning the whole thing on its head to make it look like it should have looked in the first place. Optimizing this sort of thing should start at the beginning. What is it you are trying to accomplish?  What does the data look like?  What should the output tell you?  Also, why is there no indexed data involved here at all?  It feels like the better solution would be to index one or both "sets" of this data, and ... well, it'll work better and faster.  But, I'll wait for the answers to the other questions first.  🙂 Happy Splunking, Rich         ... View more

TLDR; this is probably "hard coded" into the app.  But I put "hard coded" into quotes because it's just javascript and is editable as per the below: https://community.splunk.com/t5/Archive/7-3-Index-Selection-for-roles-does-not-show-all-indexes/m-p/499861 If that answers your question, please give them a dose of karma as well as this answer! -Rich ... View more

It could be as simple as this: index=fw ((type=utm action=blocked) OR (type=traffic action=allowed)) | stats count by srcip But probably you'll want something a bit more like index=fw ((type=utm action=blocked) OR (type=traffic action=allowed)) | stats dc(action) as has_both by srcip | search has_both>1 There's actually more possible optimization (especially if srcip is an index time field), but that's going to cut nine hundred percent off your search, I would think, and nine million percent off the search if it's actually *big*. ... View more

Disclaimer - These operating systems are no longer supported *by Microsoft* and haven't been for a while.  There are no more security updates, no more patches, no more support, no more anything.  Has the department at your company responsible for this delved into any sort of a risk assessment on this situation? Any UF you can install on them will be, by its nature, also unsupported too. Windows 7 That being said, the latest UF I could find supporting Windows 7, was 6.4.11, which really is absolutely ancient. https://docs.splunk.com/Documentation/Splunk/6.4.11/Installation/Systemrequirements (If you flip the version drop-down in the upper right to the next update, 6.5.0, it no longer lists Win7). Have you tried just installing, say, the latest in the 7.3.x series of UFs?  I mean, who knows?  Nothing you do will ever be supported (neither MS nor Splunk supports Win7 now, regardless of what version of the UF or Splunk you have), so why not just try it anyway?  I doubt 8.x will work (though you could try), but 7.x at least has a chance. Server 2008 R2- you don't mention service pack. For no service pack, I think this is the latest, another pretty ancient version: https://docs.splunk.com/Documentation/Splunk/7.0.13/Installation/Systemrequirements#Windows_operating_systems If you have it updated at least as far as you can, then you can go all the way up to https://docs.splunk.com/Documentation/Splunk/7.2.10/Installation/Systemrequirements#Windows_operating_systems Which is at least not ancient.  🙂   Happy Splunking, Rich       ... View more

So you don't get alerts because data stopped coming in? Maybe it's easiest if you rethink your needs - you don't need the alert to run continuously, what you need is for someone to know that things are broken. Then they'll know they won't be getting the alerts they're not getting because the incoming data is broken. For that task, there's a zillion ways to do this.  Searching "splunk alert data stopped coming in" in google gives you like 400 patterns to try in community/answers, and even a blog from Splunk specifically on this: https://www.splunk.com/en_us/blog/tips-and-tricks/how-to-determine-when-a-host-stops-sending-logs-to-splunk-expeditiously.html Also, if it's always supposed to send results, it should be not set to be an alert, but instead just a saved report run on a schedule.  ... View more

Right, 'transforming command' means something very specific in Splunk. https://docs.splunk.com/Splexicon:Transformingcommand You don't have any. Since you don't have any, there's a special note in the docs I linked that you may have a hard time getting it to work right. Think on it a second.  "Summary index" means "an index that summarizes".  Like, it summarizes error counts for the past 5 minutes, each 5 minutes. What you've got may be better suited to routing and filtering? https://docs.splunk.com/Documentation/Splunk/8.0.6/Forwarding/Routeandfilterdatad So, stepping back just a bit.  What problem are you trying to solve?  What is the point of the summary index?   ... View more

Could you repaste your search, only use the little code button?  The editor ate some characters. Also, there's a note right there in the middle of the "considerations for summary index searches" that warns about using summary indexes for events and not summary statistics. https://docs.splunk.com/Documentation/Splunk/8.0.6/Knowledge/Usesummaryindexing#Considerations_for_summary_index_searches It looks like your search, if the one you supplied is indeed the one populating your summary index, does not do any transforming commands and so falls into the category "getting them to work correctly can be tricky" If the search you supplied is not what's feeding the summary index, can you please post that search too? ... View more

Oh wow, that's very interesting then.  I hadn't heard of an issue like this. Now that you've clarified it a bit, I'm not quite as positive the web server IS actually running.  I mean, probably, but just that I am not longer as sure of it.  Still, my previous answer may be useful to other folks, so we'll leave it.  Just don't mark it as accepted! Have you opened a support ticket?  That's what I'd be doing, even if in tandem with this process here. Try checking index=_internal sourcetype=splunk_web_service (engine OR proxied) Or even index=_internal sourcetype=splunk_web_service (ERROR OR WARN) If those don't show much, drop even the ERROR OR WARN off it,restart and look at the whole set of splunk_web_service logs. I also tried to find a logging setting for any of the web service so we could crank it up to debug, but a quick search shows it doesn't have it's own, well named logging facility.  Hrmmm.   Secondarily, maybe look at areas like index=_internal sourcetype=splunk_web_access and see what it does when you try to access a web page for it and get that error. So to recap, 1) Open a support ticket 2) Check those logs and see if they lead anywhere useful or offer any hints. -Rich   ... View more

Oooh oooh oooh, I don't *know* the answer but I want to guess really badly. My guess is that it's trying to keep a 12 hour history of 1 minute data points, (or possibly 12 minutes of 1 second ones). I'm going to use the 12 minutes:1 second ratio below, but EITHER is exactly as valid as the other. I suspect you'll know the answer as to which one it is by looking to see if the search is scheduled to run every minute (the short side of the long period possibility), or every 12 minutes (the long side of the shorter period possibility).  So why 720 in some places, 719 in others?  I believe from inspecting it that the idea is perhaps to discount the N, since it's counted as the latestCount entry.  We first build a newAvgApiCalls, which is our current stored value for avgApiCalls from the lookup plus  1/720th of the latest calculated count. Then we create a new stdDev value by taking the square root of:   719 copies of the existing standard deviation of api calls squared,     plus 1/720th of the new count... but this gets a little squirrely in here.  I'm not convinced this is mathematically sound. And I still haven't discovered how the list is trimmed to 720, and if it's not trimmed to 720, then really this all starts getting weirder as time passes and it starts acting like a weighting function on ... the standard deviation. Interesting concept. If nothing else, I wonder about the units.  It seems like we're doing a lot of semi-circular duplicated logic inside there. If I substitute a bit.  We start with | eval newAvgApiCalls=avgApiCalls + (latestCount-avgApiCalls)/720 | eval newStdevApiCalls=sqrt(((pow(stdevApiCalls, 2)*719 + (latestCount-newAvgApiCalls)*(latestCount-avgApiCalls))/720)) OK,  let's make some names shorter and more clear and call avgApiCalls AVG, and it's a core thing we have.  And LatestCount is LATEST, it's also ground truth. newAvgApiCalls will be NEWAVG. STD and NEWSTD are our remaining, shorter, variables.  Substituting once, | eval NEWAVG= AVG + (LATEST - AVG)/720 | eval NEWSTD=sqrt(((pow(STD, 2)*719 + (LATEST-NEWAVG)*(LATEST-AVG))/720)) And then substituting once more because NEWAVG is in that second line, so let's put in what NEWAVG is (that equation): | eval NEWAVG= AVG + (LATEST - AVG)/720 | eval NEWSTD=sqrt(((pow(STD, 2)*719 + (LATEST-(AVG + (LATEST - AVG)/720))*(LATEST-AVG))/720)) And that just ... doesn't look right?  Though at least there's now a "/720" on both sides of that * in the last line, so that seems more reasonable for at least the units on that line.  What I still don't get it why you have to multiple one side by 719 and divide the other by 720, when my gut tells me maybe it should either be *719 and *1, or it should be *1 and /720 for the two sides. One side must be a rollup already.  hmm   I may dissect this farther and examine how it works on some cobbled up CSV data I can calculate manually to test with more carefully. Anyway, without regard to my curiousity which I may or may not have time to dig into much farther tomorrow some time, I hope at least my conjecture makes sense to you and fills in enough gap that it's a BIT more clear.  A bit. Anyway, happy Splunking! -Rich ... View more