Yes, the search I provided isn't intended for a whole scale, for all time, type of analysis. It's for a snapshot for operations. You are heading down the correct path in trying to leverage as much out of the *nix app as possible. There's some good stuff in there in terms of searches, tags, and event types that will make your work more simpler where it counts. If it's purely restarts and those are logged properly, you should be able to find them in the splunkd logs. To deal with size of data for aggregation, you'll need to build better filters to fit your needs. I merely gave you somewhere to start, but if you want something highly aggressive to fit your needs, you should be able to build that over time. Look for the specific types of logs you need. My search would've helped in that, if there's log gaps then likely a Splunk instance is down. Seems you'll need to look specifically for starting up logs in that case. ... View more

Your usage of maxspan vs maxpause at the moment conflict. You have two drastically different use cases, which, in turn, will require very different resource usage to complete. Best always to iterate so it's easier to track where a query isn't efficient enough, esp at scale. I specified authentication types, so that you may clear out failed auth messages that are normal, as the protocol will be to run through diff auth types until interactive or password, for example. Those extra are simply noise and would only negatively affect any statistics you try to run to capture heuristics for logging behavior wrt true login failure vs success. eventstats work very similarly to stats, only consider it as an advanced method to capture count and other stats function results but without transforming events. Consider it like an in-line function to extract count metrics, for example. The key is how to aggregate by a certain field or fields. It's a non-trivial, but highly effective function to use against your search data. I would recommend reading the splunk docs + answers to get a better feel for when/how to use it. The most important thing I can suggest in using it is to filter for your search data appropriately and then basically pivot against the fields you want to isolate specifically. There are other ways you can skin a cat, I've just found using eventstats is my method as part of your type of query here. ... View more

This is a bit more of a networking question than necessarily a Splunk administration question. Are you using straight up IPs in your output.conf? I would suggest using hostnames and editing your A records and including @harsmarvania57's input as well on the server side. Or instead use a VIP or A record pointing to multiple IPs of your indexers. His advice probably reduces the most friction possible. My advice adds to further his. Let us know your decision/progress! ... View more

Have you looked into the Splunk on Splunk app? https://splunkbase.splunk.com/app/748/ This should provide a very considerable head start on your searching performance and more! ... View more

I would strongly check your system settings like iptables as well. Very trivial to validate. ... View more

timechart only takes 1 type of field/value and runs that against time. Your requirements sound like it's nothing more than a binary UP/DOWN status here. Why not just run a count of events from the host? If there's no events within a minute period, maybe an outage (since your time window is only 15min)? index=_internal host=host01* OR host=host02* earliest=-15m | timechart span=1m count by host If you wanted to set alerts on this, you'd need to set some thresholds and throttling around the count value. Presuming scripts similar to those found in the Windows and Linux Add-ons that poll every 30s, you could generate a similar script to track this. From a pretty simple setup, this should suffice your needs hopefully! ... View more

Hi! Your search to generate this lookup table is precisely what you asked of it: Give me a table that includes job against Hour:Minute intervals I don't really understand your use case. Can you specify/clarify for the following: Why do you need to use _time? That's an internal field and I would suggest using a custom field particularly if you're performing custom time work here. Why are you trying to extract a proper date from an incomplete timestamp? This is the equivalent of trying to extraction how many millions out of a number that only in the thousands or lower. Can you use a fuller time stamp to include date? Addressing the latter question should root out any underlying notions affecting your results here. My guess is that you were running this query before 0400 local time. We'd need further clarification before providing more/better input. ... View more

Your search is good in that it captures failures followed by a successful login. But your time constraints are a bit odd. I've curated this very type of search sometime ago. I do see possibly a few design issues with your query. The first is your windowing specification. If the maxspan time window is 15min, the maxpause clause wouldn't be used. The second is any thresholding for failures before a success. Do you care about how many failures or will any do? This will need to take into consideration your systems' configuration: how many failures before lock out? Is it indefinite? The third is, make sure you know specifically what you're logging. Pay attention to the authentication methods. If it cycles through the different methods, it'll show failure on them until successful with a different method. I would consider a few things to augment your search here: Use smaller time windows to make it easier for tracking logons, i.e., X failed logon attempts within Y minutes. You can also consider multiple searches, one against smaller time windows and another for a larger, more aggregate window. Use eventstats to count the number of failures to set a threshold. Or for number of hosts attempted. Or duration. You have plenty of things to consider here, esp if you want to perform this real-time. Consider filtering for/against specific authentication methods (if ssh). I presume your logon events here are Linux or sshd based. May need to consider using calculated fields to capture the same logon success. The Splunk add-ons for Linux and Windows should have what you're looking for here as well. Please let me know if any of my input rings well with you. ... View more

To paraphrase, it sounds like you're trying to search for particular events to generate a list of sources. From this curated list, search for a different type of events, right? If so, make yourself familiar with sub-searches: http://docs.splunk.com/Documentation/Splunk/latest/Search/Aboutsubsearches Also, what precisely are you trying to look for that should belong to the first group of sources vs. what should be found from the other? Your structure would basically look like this: Subsearch query for the events: source= diameter=X Filter this to specifically the sources: source= diameter=X | fields source Insert into a different search, which includes the other field: ( [search source= diameter=X | fields source ]) temp=Y Now you have all events that fit temp=Y but only if that source has events that match diameter=X. Let me know if that works! ... View more