Hi. This is regarding Splunk 5.0.11 Universal Forwarder and Heavy Forwarder. We rebooted 2 Heavy Forwarders today (after updating glibc) and now, we are only seeing 1 of the 2 files that each of our Universal Forwarders reads and forwards. I know the data is not getting to the Indexer (or at least is not getting indexed). Is there a best practice for determining whether or not data is at least getting TO the heavy forwarder? I did try adding crcSalt= in the stanza in inputs.conf on the universal forwarder that specifies the data we're missing, just in case something cropped up. Thanks for any suggestions to help us get started with this one... ... View more

Hi folks. We have an entry in props to parse our custom datestamps (format is YYYYMMDD HHMMSS.nnn) as follows: MAX_TIMESTAMP_LOOKAHEAD = 50 NO_BINARY_CHECK = 1 SHOULD_LINEMERGE = false TIME_FORMAT = %Y%m%d %H%M%S.%3N TIME_PREFIX = ^ pulldown_type = 1 Quite often, our Splunk heavy forwarder reports that it cannot parse timestamps for these sourcetypes. What I see in the source logfiles is that sometimes, our logs do contain multiple lines (in the form of Java errors)... In that case, should we change the SHOULD_LINEMERGE from no to yes? Or do we need a far more complex regex to indicate the start of a new line? Example (seriously munged and truncated, but just to give you an idea): 0130805 160141.074 some message about an error com.stuff.api. Transactior timedout at com.stuff.exception at com.stuff.exception at com.stuff.exception at com.stuff at com.stuff at com.stuff ... View more

Hi. I see that the most recent Splunk for Exchange app is only compatible with Splunk 5.0.2. Since we are on 4.3.3 and won't be upgrading to 5 for another couple of months, I wanted to find out what the latest version is that is compatible with 4.3.3 How can I find that out? Thanks! ... View more

Hi. I have been struggling with getting to the root of some performance problems on our pool of search heads...which are two beefy servers. We do NOT see this performance issue on our other, identical site. The only difference is the users of the site and any searches they may run. When I try a splunk restart, splunkweb always hangs and the python process ultimately has to be killed manually. I have started using SoS to try to help figure this out. It shows occasional Splunkweb CPU spikes but nothing that lasts and explains the persistent slowness of our system. However, "top" shows Splunkd as the culprit, so I'm unsure where to go from there. Can anyone suggest how I might start narrowing this problem down? I have already disabled any glaringly obvious user searches that would hose the system. ... View more

Hi. We are trying the Splunk on Splunk app for the first time because one of our two environments is constantly being hammered. We have search heads in a pool and we have 4 Indexers for distributed search. Splunk version is 4.3.3. Latest S.o.S. is installed on the search heads and the SoS TA is installed on the indexers. On all servers, I have enabled the two scripted inputs. When I pull up the 20 most memory intensive searches, I get No Data returned. The Job Inspector shows the following information, but I have no idea why all of these fields are missing. I'm hoping someone has some insight! Thanks. DEBUG: Specified field(s) missing from results: '_time', 'search', 'search_head', 'user' DEBUG: [splunk1-brn] search context: user="sqig", app="sos", bs-pathname="/app/splunk/var/run/searchpeers/splunk3-head-1363707911" DEBUG: [splunk2-brn] search context: user="sqig", app="sos", bs-pathname="/app/splunk/var/run/searchpeers/splunk3-head-1363707911" DEBUG: [splunk3-brn] search context: user="sqig", app="sos", bs-pathname="/app/splunk/var/run/searchpeers/splunk3-head-1363707911" DEBUG: [splunk4-brn] search context: user="sqig", app="sos", bs-pathname="/app/splunk/var/run/searchpeers/splunk3-head-1363707911" DEBUG: [subsearch]: base lispy: [ AND index::_audit search splunk_server::splunk3-head-brn1 ] DEBUG: base lispy: [ AND index::sos sourcetype::ps ] DEBUG: search context: user="amurray", app="sos", bs-pathname="/app/splunk_mounted/etc" ... View more

Hi. We have some log data where each line starts with a timestamp that looks like this: 2012-09-28 15:44:35,302 Nothing else in the data looks anything like a timestamp. Splunk is indexing this as UTC, so it displays 4 hours earlier. The timezone on the source server is in Eastern. We are running a Splunk Universal Forwarder there, so on the Heavy Forwarder, I have the following: [my_sourcetype_here] TZ = US/Eastern For what it's worth, I also tried with [host::hostnamepattern*] Neither seem to have taken effect with newly-indexed events, despite actually restarting the Heavy forwarders! Am I missing something here? Thanks. ... View more

Hi. We recently upgraded from a 4.2 installation to 4.3.3 and a report that includes the _time field (which used to come out in epoch format) now displays the field as a formatted string. I have changed nothing with the query. It has always been a search | timchart | eval (to rename a field) | fields _time,some,other,fields That's all! No formatting, etc. The _time value used to look like this: 1341288000 Now shows up like this, including the quotation marks (this is not from the same number though!): "2012-07-24 00:00:00.000 EDT" Someone recently asked how to get epoch time, and the answer was to reference _time but as you can see. that has ceased to work for me! Anyone know about this? The documentation still says this: "The _time field is stored internally in UTC Format. It is translated to human-readable Unix time format when Splunk renders the search results (the very last step of search time event processing)." ... View more

Hi. We just upgraded from 4.2.2 to 4.3.3. We are using search head pooling, so we followed the specific instructions for dealing with that situation (ie, unpool, upgrade each head, repool). Now, it seems that Views and Saved Searches by some of our users are showing up as having no owner. I checked and it looks to me like the user's directory exists in $SHARED/etc/users and (as you might expect) not in $SPLUNK_HOME/etc/users Has anyone else run into this? I did make a backup of everything before the upgrade, so if the upgrade clobbered some critical files I'm not aware of, I could replace them, I just don't even know where to start! ... View more

Hi. Been trying to work this one out for hours... I'm close!!! We are Splunking data such that each Host has a field "SomeText" which is some arbitrary string, and that string may be repeated on that host any number of times. It may also appear on other hosts... Basically, think of something like a syslog file... your crond message can be any number of different strings. Let's say that Host1 has the following strings: "The quick brown fox" shows up 5 times "jumps over the" shows up 2 times "lazy dog" shows up 10 times "My dog has fleas" shows up 2 times "So does yours" also shows up 2 times I want a chart that shows me: Host1 10 "lazy dog" 5 "The quick brown fox" 2 "jumps over the" 2 "My dog has fleas" 2 "So does yours" But what I GET is this: Host1 1 "lazy dog" 2 "The quick brown fox" 5 "jumps over the" "My dog has fleas" "So does yours" (I think the string column is actually sorted alphabetically). This is a mockup of the search I'm running, with field names obviously simplified: index=myindex earliest=-24h | stats count(SomeText) as textCount by SomeText host | stats values(textCount) as Count,values(SomeText) as "Text" by host What am I missing? How can I marry up the # of times a message appears with that message? Thanks for any ideas. ... View more