Hey guys, Doesn't seem like many people have had problems with the Splunk Web Service hanging on them, but this is somewhat similar to: http://answers.splunk.com/questions/1468/splunkweb-crashes-although-splunkd-is-still-running-is-this-a-known-problem Basically, I have been testing my Foundstone Vulnerability Scanner on my splunk indexers and search head and it looks like at least one of servers has its Splunk Web Service die. Port is reachable via telnet, and splunk status states that the service is up, but the site itself is unreachable. I simulated the situation and tailed the logs and see these types of errors: splunkd.log 06-22-2010 20:42:42.816 ERROR NetUtils - SSL_ERROR_SSL in SSL_write. nbytes=-1, Error = error:140D00CF:SSL routines:SSL_write:protocol is shutdown 06-22-2010 20:42:43.210 ERROR TcpInputFd - SSL Error = error:1407609C:SSL routines:SSL23_GET_CLIENT_HELLO:http request 06-22-2010 20:42:43.210 ERROR TcpInputFd - ACCEPT_RESULT=-1 VERIFY_RESULT=0 06-22-2010 20:42:43.210 ERROR TcpInputFd - SSL Error for fd from HOST:$foundstone_scanner$, IP:$foundstone_IP$, PORT:5927 06-22-2010 20:42:43.365 ERROR TcpInputFd - SSL Error = error:1407609C:SSL routines:SSL23_GET_CLIENT_HELLO:http request 06-22-2010 20:42:43.365 ERROR TcpInputFd - ACCEPT_RESULT=-1 VERIFY_RESULT=0 06-22-2010 20:42:43.365 ERROR TcpInputFd - SSL Error for fd from HOST:$foundstone_scanner$, IP:$foundstone_IP$, PORT:5939 06-22-2010 20:42:44.116 ERROR TcpInputFd - SSL Error = error:1407609C:SSL routines:SSL23_GET_CLIENT_HELLO:http request 06-22-2010 20:42:44.116 ERROR TcpInputFd - ACCEPT_RESULT=-1 VERIFY_RESULT=0 06-22-2010 20:45:11.376 INFO TailingProcessor - File descriptor cache is full (64), trimming... 06-22-2010 20:45:17.397 INFO TailingProcessor - File descriptor cache is full (64), trimming... 06-22-2010 20:45:19.516 INFO TailingProcessor - File descriptor cache is full (64), trimming... 06-22-2010 20:46:22.830 INFO TailingProcessor - File descriptor cache is full (64), trimming... web_service.log This is the error message that appears right before it hangs: 2010-06-18 19:05:05,842 ERROR [4c1798223163010d0] root:120 - ENGINE: Error in HTTP server: shutting down Traceback (most recent call last): File "/opt/splunk/lib/python2.6/site-packages/cherrypy/process/servers.py", line 73, in _start_http_thread File "/opt/splunk/lib/python2.6/site-packages/cherrypy/wsgiserver/init.py", line 1662, in start self.tick() File "/opt/splunk/lib/python2.6/site-packages/cherrypy/wsgiserver/init.py", line 717, in tick s, addr = self.socket.accept() File "/opt/splunk/lib/python2.6/ssl.py", line 317, in accept newsock, addr = socket.accept(self) File "/opt/splunk/lib/python2.6/socket.py", line 195, in accept error: [Errno 24] Too many open files Anyone ever have the same issue? I also ran splunk diag so that I can open support ticket but just wanted to see if anyone else has ever had this problem. Let me know. Thanks Guys! Brian ... View more

Hey guys, We were testing out deployment configs for one of our deployment Light Forwarder apps and happened to delete a file from our deployment client Light Forwarder app default directory ($Splunk_Home/etc/apps/myapp/default). I would assume that on the next phone home the deployment server would have noticed that the deployment client was missing the deleted file and add it back but that does not seem to be the case. From a network perspective, the Client is phoning home to the server fine but is not picking up the default file that was deleted. Any thoughts? Anyone run into this? Let me know. Thanks. Brian ... View more

For some reason, looks like 2-3 of my indexes have stopped indexing. The monitor point to the indexes is pointed to directories storing syslog-ng files, and all my indexes currently indexing are setup the exact same way. I tried looking in my indexer's splunk logs but can't find anything helpful. Has this ever happened to anyone before? What are typical reasons an indexer would stop indexing? networking is not an issue since the indexer is indexing local files. I see that splunk service is working fine, and my other indexes are indexing just fine... I see splunk ports are open... so what am I missing here? I am super stumped on this one. Appreciate a point in the right direction. Let me know if you need more details to help arrive at the cause of the issue. Thanks guys. Brian ... View more

Hey guys, We are monitoring 2 specific CSV Log files on one indexer. I setup the appropriate custom field extractions for the CSV files in the props.conf and transform.conf files for both the indexer and the search head. If I search directly on the indexer it works fine. However, if when I try to search the same files through the search head I am not able to see the custom field extractions I have created. Any thoughts? Here is what I have for the props.conf file for both the indexer and the search head: PROPS.CONF [palo_alto_traffic] REPORT-paextract = paloalto_traffic_extractions KV_MODE = none CHECK_FOR_HEADER = true TRANSFORMS-NoHeader = NoHeader_paloalto [palo_alto_threat] REPORT-paextract = paloalto_threat_extractions KV_MODE = none CHECK_FOR_HEADER = true TRANSFORMS-NoHeader = NoHeader_paloalto and here is the contents of the transforms.conf file for both the search head and the indexer: TRANSFORMS.CONF [paloalto_traffic_extractions] DELIMS = "," FIELDS = "Domain" , "Receive_Time" , "Serial_Number" , "Threat_Content_Type" , "Config_Version" , "Generate_Time" , "Source_address" , "Destination_address" , "NAT_Source_IP" , "NAT_Destination_IP" , "Rule" , "Source_User" , "Destination_User" , "Application" , "Virtual_System" , "Source_Zone", "Destination_Zone" , "Inbound_Interface", "Outbound_Interface" , "Log_Setting" , "Time_Logged" , "Session_ID" , "Repeat_Count" , "Source_Port" , "Destination_Port" , "NAT_Source_Port" , "NAT_Destination_Port" , "Flags" , "IP_Protocol" , "Action" , "Bytes" , "Bytes_Sent" , "Bytes_Received" , "Packets" , "Start_Time" , "Elapsed_Time_Sec" , "Category" , "Padding" [paloalto_threat_extractions] DELIMS = "," FIELDS = "Domain" , "Receive_Time" , "Serial_Number" , "Type" , "Threat_Content_Type" , "Config_Version" , "Generate_Time" , "Source_address" , "Destination_address" , "NAT_Source_IP" , "NAT_Destination_IP" , "Rule" , "Source_User" , "Destination_User" , "Application" , "Virtual_System" , "Source_Zone" , "Destination_Zone" , "Inbound_Interface" , "Outbound_Interface" , "Log_Setting" , "Time_Logged" , "Session_ID" , "Repeat_Count" , "Source_Port" , "Destination_Port" , "NAT_Source_Port" , "NAT_Destination_Port" , "Flags" , "IP_Protocol" , "Action" , "URL" , "Threat_Content_Name" , "Category" , "Severity" , "Direction" [NoHeader_paloalto] REGEX = Domain,Receive Time,Serial #,Type,Threat/Content Type, ... DEST_KEY = queue FORMAT = nullQueue Let me know. Thanks. Brian ... View more

Hi, we are currently testing a Palo Alto app sec firewall and are sending some test logs over to the central indexer in CSV format. Can splunk read CSV files? The splunkd.log file is giving me somewhat of an error stating the following: 05-25-2010 14:21:33.678 ERROR TailingProcessor - Ignoring path due to: File will not be read, is too small to match seekptr checksum (file=PATHTOMYFILES). Last time we saw this initcrc, filename was different. You may wish to use a CRC salt on this source. Consult the documentation or contact Splunk Support for more info. Let me know. Thanks! Brian ... View more