Getting Data In
Highlighted

How to troubleshoot why I can see network traffic, but no data is being indexed in Splunk from our Cisco FWSM?

Path Finder

Hi All,

I have been having issues with getting logs into splunk from our cisco fwsm. When I open up wireshark I can see network traffic coming in but it does not hit any index. To prove this theory I searched all of the potential indexes plus looked up via IP address. I am not sure where to start to troubleshoot. Does anyone know about debug logs that I can review? I have installed and configured the app as per guidelines written by cisco so I am pretty sure that is ok. Any help would be appreciated.

0 Karma
Highlighted

Re: How to troubleshoot why I can see network traffic, but no data is being indexed in Splunk from our Cisco FWSM?

Path Finder

Also I have added the following as a catchall just in case and it is still not working. I have specified both main and ciscocore as the index and no luck.

Path Splunk\etc\system\local

props.conf
[host::x.x.x.x]
TRANSFORMS-index=sendFirewallLogs

[host::x.x.x.x]
TRANSFORMS-index=sendFirewallLogs2

[sendFirewallLogs]
REGEX=.
DESTKEY=MetaData:Index
FORMAT=main
WRITE_META=true

[sendFirewallLogs2]
REGEX=.
DESTKEY=MetaData:Index
FORMAT=main
WRITE_META=true

0 Karma
Highlighted

Re: How to troubleshoot why I can see network traffic, but no data is being indexed in Splunk from our Cisco FWSM?

Motivator

Could you paste the contents of your inputs.conf? You should have a stanza for UDP port 514 or any other port you chose.

0 Karma
Highlighted

Re: How to troubleshoot why I can see network traffic, but no data is being indexed in Splunk from our Cisco FWSM?

Path Finder

My inputs.conf is in a default state.

I can confirm that there is a setting in data inputs and receiving for port 514 and a netstat -ano | findstr "514" shows the UDP port.

0 Karma
Highlighted

Re: How to troubleshoot why I can see network traffic, but no data is being indexed in Splunk from our Cisco FWSM?

Motivator

And no data for FWSM when you search:

index=*

Over all time? All time because we want to rule out any timestamp issues.

Just a wild guess, but could it be the windows firewall?

0 Karma
Highlighted

Re: How to troubleshoot why I can see network traffic, but no data is being indexed in Splunk from our Cisco FWSM?

Path Finder

Nah its cisco fwsm. Yeah I tried the index=* and there was nothing in the logs. So for a little more background it was working previously but then stopped one day. I am not sure why though. So historic data is still available but not current data.

Thanks for your help. Splunk answers was my last ditch effort as I have gone through most of the articles on splunk answers but with no luck :(.

The old bucket it was going to was 'main'

0 Karma
Highlighted

Re: How to troubleshoot why I can see network traffic, but no data is being indexed in Splunk from our Cisco FWSM?

Motivator

What I meant was if it could be the Windows firewall blocking inbound connections on UDP port 514, or perhaps another local firewall?

I assume the traffic you saw in your wireshark capture was coming in on UDP 514? 100% sure no one changed it to TCP on the FWSM?

0 Karma
Highlighted

Re: How to troubleshoot why I can see network traffic, but no data is being indexed in Splunk from our Cisco FWSM?

Path Finder

Yeah 100% its on UDP and firewall is not interfering :). As there is other syslog data coming in on the same port. Is there any debug logs that I can enable to see why an index is not receiving data?

0 Karma
Highlighted

Re: How to troubleshoot why I can see network traffic, but no data is being indexed in Splunk from our Cisco FWSM?

Motivator

Not too sure about what log this would go in, but splunkd.log and splunkdstdout.log and splunkdstderr.log would be my guesses.

0 Karma
Highlighted

Re: How to troubleshoot why I can see network traffic, but no data is being indexed in Splunk from our Cisco FWSM?

Path Finder

So I started splunk in debug mode and noticed the following event with the correct source IP of my firewall. So it seems that the data is being accepted but not sure where though.

03-20-2015 08:34:30.821 DEBUG UDPInputProcessor - event=data from=x.x.x.x status=accepted

0 Karma