topic Re: How to troubleshoot why I can see network traffic, but no data is being indexed in Splunk from our Cisco FWSM? in Getting Data In

How to troubleshoot why I can see network traffic, but no data is being indexed in Splunk from our Cisco FWSM?

domenico_perre — Wed, 18 Mar 2015 00:09:13 GMT

Hi All,

I have been having issues with getting logs into splunk from our cisco fwsm. When I open up wireshark I can see network traffic coming in but it does not hit any index. To prove this theory I searched all of the potential indexes plus looked up via IP address. I am not sure where to start to troubleshoot. Does anyone know about debug logs that I can review? I have installed and configured the app as per guidelines written by cisco so I am pretty sure that is ok. Any help would be appreciated.

Re: How to troubleshoot why I can see network traffic, but no data is being indexed in Splunk from our Cisco FWSM?

domenico_perre — Mon, 28 Sep 2020 19:10:56 GMT

Also I have added the following as a catchall just in case and it is still not working. I have specified both main and ciscocore as the index and no luck.

Path Splunk\etc\system\local

props.conf
[host::x.x.x.x]
TRANSFORMS-index=sendFirewallLogs

[host::x.x.x.x]
TRANSFORMS-index=sendFirewallLogs2

[sendFirewallLogs]
REGEX=.
DEST_KEY=_MetaData:Index
FORMAT=main
WRITE_META=true

[sendFirewallLogs2]
REGEX=.
DEST_KEY=_MetaData:Index
FORMAT=main
WRITE_META=true

Re: How to troubleshoot why I can see network traffic, but no data is being indexed in Splunk from our Cisco FWSM?

mikaelbje — Wed, 18 Mar 2015 06:56:03 GMT

Could you paste the contents of your inputs.conf? You should have a stanza for UDP port 514 or any other port you chose.

Re: How to troubleshoot why I can see network traffic, but no data is being indexed in Splunk from our Cisco FWSM?

domenico_perre — Thu, 19 Mar 2015 00:49:25 GMT

My inputs.conf is in a default state.

I can confirm that there is a setting in data inputs and receiving for port 514 and a netstat -ano | findstr "514" shows the UDP port.

Re: How to troubleshoot why I can see network traffic, but no data is being indexed in Splunk from our Cisco FWSM?

mikaelbje — Thu, 19 Mar 2015 02:22:05 GMT

And no data for FWSM when you search:

index=*

Over all time? All time because we want to rule out any timestamp issues.

Just a wild guess, but could it be the windows firewall?

Re: How to troubleshoot why I can see network traffic, but no data is being indexed in Splunk from our Cisco FWSM?

domenico_perre — Thu, 19 Mar 2015 03:08:52 GMT

Nah its cisco fwsm. Yeah I tried the index=* and there was nothing in the logs. So for a little more background it was working previously but then stopped one day. I am not sure why though. So historic data is still available but not current data.

Thanks for your help. Splunk answers was my last ditch effort as I have gone through most of the articles on splunk answers but with no luck :(.

The old bucket it was going to was 'main'

Re: How to troubleshoot why I can see network traffic, but no data is being indexed in Splunk from our Cisco FWSM?

mikaelbje — Thu, 19 Mar 2015 06:07:12 GMT

What I meant was if it could be the Windows firewall blocking inbound connections on UDP port 514, or perhaps another local firewall?

I assume the traffic you saw in your wireshark capture was coming in on UDP 514? 100% sure no one changed it to TCP on the FWSM?

Re: How to troubleshoot why I can see network traffic, but no data is being indexed in Splunk from our Cisco FWSM?

domenico_perre — Thu, 19 Mar 2015 06:11:42 GMT

Yeah 100% its on UDP and firewall is not interfering :). As there is other syslog data coming in on the same port. Is there any debug logs that I can enable to see why an index is not receiving data?

Re: How to troubleshoot why I can see network traffic, but no data is being indexed in Splunk from our Cisco FWSM?

mikaelbje — Mon, 28 Sep 2020 19:15:01 GMT

Not too sure about what log this would go in, but splunkd.log and splunkd_stdout.log and splunkd_stderr.log would be my guesses.

Re: How to troubleshoot why I can see network traffic, but no data is being indexed in Splunk from our Cisco FWSM?

domenico_perre — Thu, 19 Mar 2015 22:24:16 GMT

So I started splunk in debug mode and noticed the following event with the correct source IP of my firewall. So it seems that the data is being accepted but not sure where though.

03-20-2015 08:34:30.821 DEBUG UDPInputProcessor - event=data from=x.x.x.x status=accepted

Re: How to troubleshoot why I can see network traffic, but no data is being indexed in Splunk from our Cisco FWSM?

domenico_perre — Fri, 20 Mar 2015 00:08:07 GMT

So I am going to answer my own question which is a good thing I guess. The way I understand the issue was the following:

In Settings | Data Inputs | UDP | 514 the sourcetype was forced to syslog.
The transforms that are part of the cisco ASA /fwsm app install state that the source type is a cisco fwsm.
I had previous data in the main index that was specified as sourcetype cisco:fwsm
The two transforms, 'syslog' (that is forced on the Data inputs) and 'cisco:fwsm' (that is forced from the cisco app) were competing against each other and thus the firewall event was dropped.

How did I resolve this. I removed the source 514 from the GUI.
In inputs.conf I added the following lines.

[udp://514]
connection_host = ip

Restarted splunk and it started working. So the issue which did not rear its ugly head in any debug logs was because of competing transforms I believe. That moment when you hit the Splunk matrix and everything just works!!!

Re: How to troubleshoot why I can see network traffic, but no data is being indexed in Splunk from our Cisco FWSM?

mikaelbje — Fri, 20 Mar 2015 02:40:39 GMT

Could you try adding the following to the udp://514 stanza:

sourcetype=syslog

This should work as I have more than a dozen customer installs where the sourcetype is set as syslog, and add-ons such as Cisco ASA add-on and Cisco Networks add-on change the sourcetype based on a transform. This works out of the box with a lot of apps.

Make sure you do not set a value for the source field, only sourcetype.

Re: How to troubleshoot why I can see network traffic, but no data is being indexed in Splunk from our Cisco FWSM?

domenico_perre — Fri, 20 Mar 2015 03:02:45 GMT

As soon as I put that field in under the stanza and restarted the splunk service, logs stopped working.

Re: How to troubleshoot why I can see network traffic, but no data is being indexed in Splunk from our Cisco FWSM?

mikaelbje — Fri, 20 Mar 2015 05:41:39 GMT

Could you paste the output of the following commands?

Go to your Splunk working dir/bin

./splunk cmd btool props list syslog --debug
./splunk cmd btool inputs list --debug