Hi Splunkers, My indexers are running Splunk Enterprise v6.5.3. I recently upgraded a "test" Universal Forwarder in my environment to v6.6.5, and I'm no longer getting logs going to my indexers from this "test" UF after the upgrade. I'm seeing a bunch of these errors before the logs stopped: WARN SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read server hello A', alert_description='handshake failure'. Is this an SSL or cipherSuite incompatibility issue between the two different versions of Splunk? Is there a workaround to get the test forwarder sending logs again, or do I have no choice but to either 1. downgrade the forwarder -OR- 2. upgrade my indexers? Thank you! ... View more

Hi Splunkers! I have a Splunk distributed deployment. One of my customers has a separate Splunk distributed deployment on the same network as mine. All of their Splunk Enterprise servers run on Linux. This customer has recently asked me to ingest basic Linux security logs (things like /var/log/secure) from all of their Splunk Enterprise servers into my Splunk indexers. This is to maintain audit compliance. Since these servers are full-blown Splunk Enterprise servers, and not just Universal Forwarders, I'm having some trouble getting logs to come over to my side correctly. They are already indexing Linux security logs on their side via a local inputs.conf in the Splunk TA for NIX. To get the same logs to my indexers, I'm not sure if this is a matter of putting an outputs.conf on their servers (or modifying an existing outputs.conf on their servers) to also point to my indexers, and then adding a _TCP_ROUTING entry in their existing inputs.conf (the one in the TA for NIX). Also, their linux logs index has a different name than mine. Do I need to set up something on my side (props? transforms?) to do the index translation when the logs arrive on my side? Thank you in advance! ... View more

Hi Splunkers, My program is considering adding 600 more Linux UF endpoints to our current Splunk deployment (we have ~450 total UF endpoints now), and they're asking for a "wish list" of resources to support the additional volume. I have a pretty good idea of my licensing needs, and I've been using the Splunk online sizing tool to figure out how much additional disk capacity we need (based on our retention policies). Is there also a good sizing tool or document out there to help me figure out whether I need to increase RAM/CPU on my indexers, and possibly add another indexer? (and maybe add another deployment server) Just FYI - I currently have a 2 indexer cluster. Each indexer has 16 cores, 31 GB RAM ... View more

Hi Splunkers! I’d like to pick your brain to see if you know of 3-5 key windows event log events to monitor that would indicate a machine that has crashed or is having trouble with a particular component (application, hardware, driver, etc). I’m working on a set of alerts in Splunk for my program to assist with maintaining their uptime SLAs. I’m looking to search in Splunk for a simple text string, event id, error code, or pattern that would indicate that a system has gone down or is degraded (i.e. something is failing). I’ve done some research. Here’s what I’ve got so far: System Log, Event ID: 41, Source: Microsoft-Windows-Kernel-Power Description: The system has rebooted without cleanly shutting down first. The kernel power event ID 41 error occurs when the computer is shut down, or it restarts unexpectedly An unexpected reboot error appears in the log when the system fails to shut down and restart gracefully. A likely cause of this error is that the operating system stopped responding and crashed, or the server lost power. System Log, Event ID: 6008, Source: EventLog Description: “The previous system shutdown at on was unexpected.” This event id will let you know that the system started after it was not shut down properly. System Log, Event ID: 18, Source: Microsoft-Windows-WHEA-Logger Description: “A fatal hardware error has occurred.” This error indicates that there is a hardware problem Application Log, Level: Error, Source: Application Error Description: Tracking applications that have crashed or faulted on the system System Log, Event ID: 7000, Source: Service Control Manager Description: “The service failed to start due to the following error: ”. This error is logged when a service fails to start normally. Any thoughts? ... View more