I have been having issues with getting logs into splunk from our cisco fwsm. When I open up wireshark I can see network traffic coming in but it does not hit any index. To prove this theory I searched all of the potential indexes plus looked up via IP address. I am not sure where to start to troubleshoot. Does anyone know about debug logs that I can review? I have installed and configured the app as per guidelines written by cisco so I am pretty sure that is ok. Any help would be appreciated.
Also I have added the following as a catchall just in case and it is still not working. I have specified both main and ciscocore as the index and no luck.
Could you paste the contents of your inputs.conf? You should have a stanza for UDP port 514 or any other port you chose.
My inputs.conf is in a default state.
I can confirm that there is a setting in data inputs and receiving for port 514 and a netstat -ano | findstr "514" shows the UDP port.
And no data for FWSM when you search:
Over all time? All time because we want to rule out any timestamp issues.
Just a wild guess, but could it be the windows firewall?
Nah its cisco fwsm. Yeah I tried the index=* and there was nothing in the logs. So for a little more background it was working previously but then stopped one day. I am not sure why though. So historic data is still available but not current data.
Thanks for your help. Splunk answers was my last ditch effort as I have gone through most of the articles on splunk answers but with no luck :(.
The old bucket it was going to was 'main'
What I meant was if it could be the Windows firewall blocking inbound connections on UDP port 514, or perhaps another local firewall?
I assume the traffic you saw in your wireshark capture was coming in on UDP 514? 100% sure no one changed it to TCP on the FWSM?
Yeah 100% its on UDP and firewall is not interfering :). As there is other syslog data coming in on the same port. Is there any debug logs that I can enable to see why an index is not receiving data?
Not too sure about what log this would go in, but splunkd.log and splunkdstdout.log and splunkdstderr.log would be my guesses.
So I started splunk in debug mode and noticed the following event with the correct source IP of my firewall. So it seems that the data is being accepted but not sure where though.
03-20-2015 08:34:30.821 DEBUG UDPInputProcessor - event=data from=x.x.x.x status=accepted