Sounds like a networking problem. The forwarder is saying that it cannot open a socket to your 10.x.x.x host on 9997. Possibilities include: Firewall (unlikely, connection refused is not how firewalls normally behave) Routing: perhaps 10.x.x.x means something different on that network? port 9997 is not open: Maybe splunk is not running on that host right now Maybe splunk is not configured to receive data on 9997 on that host You can probably rapidly rule out the client behavior by trying to telnet to that port from those systems. Likely, you will get the same error. ... View more

IN addition, your rex command is trying to capture a space, while your extraction is not trying to capture a space. I'm not sure if you want one or not. If you do, that would probably be REGEX = \s([a-z][a-z]-[a-z][a-z][a-z][0-9][0-9]) ... View more

Note that reparsing already parsed events means that they will be timestamped and possibly merged a second time, with possibly worse results than the first time. There are also some performance costs with processing all data again even if it is already processed. ... View more

Most likely this is one of a category of problems where the URL generated by the browser to splunkweb, or by splunkweb to splunkd is too large to work. We've fixed some related issues in the 4.1.5 to 4.2 stretch, so depending upon specifics it might work in a later revision. In short, that's a bug not a configuration. A dumb workaround might be to use |outputcsv in your search, and then go fetch the file out of the dispatch directory on the splunk server. Obviously thats not a scalable solution, but perhaps it gets you out of a bind for the moment. Currently, the only good path for customers to report bugs is via support. Please do so, especially with some hints as to what the set of fields looks like. I'm suspicous that it has to do partly with field name length as well as quantity. IF you can show it cuts off at exactly 500 no matter what, that is useful info! ... View more

The UNIX app was generally designed to not require root. This made solving a lot of problems much harder. It's possible an oversight may have crept in somewhere however. ... View more

The mount method of config availability is probably not worth considering until you have fairly large amounts of searchtime data, such as larger lookups. (Our replication algorithm is currently -- 4.1/4.2 -- a bit naive, so a frequently changing small file will cause resending of a static large file.) This may be the common case for large, many indexer environments, I don't personally know. ... View more

More practically, this means that changes to searchtime configuration may remain stale for a relatively short window of time. For example, if the way a field is extracted is modified, the 4.1 search node (indexer) may not perform searches with this new field extraction until a small number of minutes later. ... View more

In 4.1.3, we made it so the events viewer had a configurable max eventsize ceiling (so we dont crash the browser). This is mentioned here: http://answers.splunk.com/questions/559/how-do-you-tweak-splunk-to-display-an-event-that-is-more-than-500-lines-long I think show source is still limited, as of 4.2, to 100 lines per event. ... View more

Weird that you cant comment. Looks like this is an as yet unfixed bug. Filing but would be ideal to have customers behind it if you still care (just open a case with www.splunk.com/support) ... View more

http://www.splunk.com/base/Documentation/latest/User/ChangeTheTimeRangeOfYourSearch#Specify_relative_time_ranges_in_your_search Maybe: ((index=larger_general_index earliest=-5m) OR (index=categorized_rare_events latest=-5m)) interesting terms ... View more

Its the default because you can say server = server1, server2 and then you are balancing. I think you actually turn off a bunch of logic to sense servers in shutting-down state with this change, which means you get slightly worse behavior on server restart. In any event it certainly narrows the problem down, its the tester socket connections to see if the server is still accepting incoming connections (isnt shutting down, or backed up). ... View more

We ship a tar of the files for a given platform that comes up and works. In the Universal Forwarder, we don't even create per-machine hostname settings etc. IMO, unpack the tar, change what you need, make new tar to deploy. ... View more

The 'double' timestamp may just be the splunk UI, then. Maybe a picture would help. As for getting the hostname from the data, this is a matter of your monitor stanza configuration. Be sure to set it to syslog. ... View more

You could also use the custom condition alert search and have that look at your maintenance window to control supression. This assumes you can express the window in terms of a splunk search. ... View more

We should get the matching crashlog (and on windows DMP) files and get a bug opened. If you're hoping for a guess, you could ask Dr Z if he has any idea. ... View more

I recommend showing more of the line and stating what file. Most likely this is the line from sendemail.py in python.log, and is probably the message from the mailserver. Maybe ask the maiserver admins if they see anything helpful on their end? ... View more

netwrkr gave the useful answer. The thing is the spec files are generated dynamically for each release for each build. This means we dont actually keep them. It also means they're not convenient for third-party use. My personal recommendation is to use the tar for automation purposes. It will likely be less trouble in the end. You may have strong reasons to feel otherwise, of course. ... View more

Not sure if you're saying searches are slow, or indexing is slow. If it's searches, you generally want to look at the specific search. If it's indexing, configuring timestamping and event boundary detection are the biggest boosts. Its possible you're hitting machine constraints. Normal system analysis is very helpful to define the problem. Note that Splunk indexes all events by keywords, so there's usually no need to add special indexed fields, for most searches and most data patterns they only hurt. ... View more

The normal alert mechanisms will get the 'results' stream of a search, so the summarization will hide the events stream from the alerts. The simplest approach would be to run it again, but differently, to get the events. Probably its slow or you wouldn't be asking this. The smart way would probably be to have a second search run using the prior search's artifacts. However, I don't know how to have two scheduled jobs communicate such a thing. One interesting hackable feature is the 'custom condition search' which does run precisely this way. You could have for example a search which produces your events, then you hack the summarization into the 'alert condition' follow-on (warning, 10k default event limit), have that search emit at least one result, which will fire to cause the raw events from the original search to be sent. A particularly evil method might be to use a custom alert script, and given the path to the results file, modify it to reach the events file, and grab those. ... View more

In the search actions, try "Get Link to results"... ... View more

The variable in etc/system/default/server.conf has always been $HOSTNAME-$USERNAME or whatever like that. Usually this has never mattered, because it got set to servername in install in so-called "first time run. However, there was a goal to do mass rollouts of the Universal Forwarder with no config tweaks at all, pre-installed etc, so you are now seeing these variable expansions take place. I was never sure of the goal of having $USER in there. I suspect it was for development purposes at HQ. If it causes any operational difficulties/annoyances we should get a bug or at least a ticket filed to discuss. ... View more

Do these crash events have any telltales? There could be crashes for other reasons. ... View more

Splunk stores the log contents. The log files are not needed by splunk at search time. ... View more

Fundamentally, you are asking for Splunk to sort a very large amount of data along an axis by which it isn't already sorted. I think you aren't interested in all the results, just the outliers, so there's a simple hack to get started: index="production" sourcetype="access" timetaken > 300 | sort -timetaken |head 10 If you have any events at all in the data which arent candidates, you'd want to further narrow the initial search, eg index="production" sourcetype="access" wanted NOT unwanted timetaken > 300 | sort -timetaken | head 10 A fancier way might be to do a subsearch (or feed a lookup by searching) over a short interval take perc95 of the timetaken, and then clip the 24 hour period to timetaken > that value. This approach could be further combined with the summary way if needed. ... View more