I would think using a summary index would always be faster. ... View more

Did not realize you were running this on Windows. Sorry as that is not my area of expertise. Hopefully someone else will chime in shortly. In the meantime I would review all the log files in the Splunk log directory $SPLUNK_HOME/var/log/splunk/ ... View more

As long as the location of the log files doesn't change, any CPanel updates pushed shouldn't really have an impact on Splunk and forwarding. What could be an issue, is if they changed the log format - ie. you are currently parsing Apache logs based on the CLF and they push out a change that implements a different format. In my many (10+) years of experience with CPanel, I've never seen them do that before. ... View more

Are you running SELINUX? What does 'ulimit -a' look like? 'grep ulimit splunkd.log' ... View more

This seems like it could be a bug. Anything in the splunk logs? ... View more

This isn't too hard. Using the transaction command along with 'startswith' and 'endswith' you want to search on transactions that aren't closed and then output those unclosed (or evicted) transactions "|search closed_txn=0" keepevicted= Description: Whether to output evicted transactions. Evicted transactions are events that do NOT match the transaction parameters; for example, the time range is wrong, or the "startswith" or "endswith" requirements are missing. Evicted transactions can be distinguished from non-evicted transactions by checking the value of the 'closed_txn' field, which is set to '0' for evicted transactions and '1' for closed ones. A transaction is evicted from memory when the memory limitations are reached. HTH ... View more

The v4.2 Interactive Field Extractor sucks IMO. Highly recommend using something like RegEx Buddy or RegEx Magic. They are cheap apps but really make short work of regex's. ... View more

I think because the time from the log file was too far in the past Splunk discards it and instead uses "index" time - that is the time the event was indexed. I seem to recall reading this in the docs from previous version but can no longer find such a reference. Here is a good article though that may help you understand things a bit better - http://www.splunk.com/base/Documentation/4.2/Data/HowSplunkextractstimestamps [edit] - found the info here about timestamps in the past / future http://www.splunk.com/base/Documentation/4.2/Data/Configuretimestamprecognition MAX_DAYS_AGO = Specifies the maximum number of days in the past, from the current date, that an extracted date can be valid. For example, if MAX_DAYS_AGO = 10 then Splunk ignores dates older than 10 days from the current date. Default is 2000. Note: If you have data that is more than 2000 days old, increase this setting. MAX_DAYS_HENCE = Specifies the maximum number of days in the future from the current date that an extracted date can be valid. For example, if MAX_DAYS_HENCE = 3, dates that are more than 3 days in the future are ignored. False positives are less likely with a tighter window. If your servers have the wrong date set or are in a timezone that is one day ahead, set this value to at least 3. Defaults to 2. This allows timestamp extractions that are up to a day in the future. ... View more

Anything different about the logs associated with the incorrect time? I assume Splunk is using 'index' time versus parsing the time from the logged event? ... View more

Splunk can do just about anything provided you know how to write the search query. Take a look at the 'transaction' command - I believe it is what you seek - http://www.splunk.com/base/Documentation/latest/SearchReference/Transaction ... View more

Not that I'm aware of. Would the 'highlight' command help? ... View more

Also running CentOS 5.5 x64. No issues. ... View more

depends where you installed the tgz at. as I've never used the rpm I don't know where it installs splunk. given that everything you need is both in the tarball and rpm why switch? ... View more

It appears in 4.2 you can 'throttle' alerts - more info here - http://www.splunk.com/base/Documentation/latest/User/Alertusecases ... View more

Marking my answer as helpful is always appreciated 🙂 ... View more

Kevin makes good points above. I will also add that having Splunk read in gzip'd files is pretty resource intensive. gzip is not multi-threaded and depending on the number of CPU's in your server, and the number of log files it is simultaneously uncompressing, that alone could chew up a lot of your processor(s). ... View more

The way I suggested above is to group at search time. Splunk has a nice document which details how to extract fields here - http://www.splunk.com/base/Documentation/latest/User/InteractiveFieldExtractionExample ... View more

I was just using auid as an example. Grouping based on PID and/or SID probably makes the most sense. ... View more

A couple of quick thoughts - are you reading in compressed (gz) files? Is your server possibly undersized for the amount of data you are ingesting? Have you looked at the "Status" charts available within the search UI? They contain a lot of useful information that may help you track down whats making things slow. ... View more

Could you name the log file with the associated date / time value at the beginning rather than changing it afterwards? ... View more