Why tcpdump says Syslog traffic from Cisco ASA is reaching the interface, but nothing shows up when searching the index?

[rblalock]

I have a Cisco ASA sending syslog data to my Splunk server. When I search for the ip address of the ASA in the Search & Reporting app, there is nothing. But if I ssh into the server and look for the traffic with tcpdump, there is syslog traffic from that server arriving at the interface. Why does the data not show up when I search the index? I have several other ASAs that are logging syslog data successfully.

[slebbie_splunk]

I would agree with wingfieldj, check the configuration files for the ASA forwarder and see where the logs are being forwarded too, making sure it matches the indexer you are querying.

Another random guess is that make the bucket size maybe maxed/capped out, if your Splunk indexer can only hold a certain amount of data, the forwarder may send logs all day but based may either be reaching the indexer much delayed. I highly doubt this because you should have at least some sort of data indexed from the ASA but just giving a wide shot thought.

But check the configs on both the forwarder and the indexer and if it matches, then look into the dropped data possibility.

[EdBruce]

I keep seeing this or similar issues but have yet to find an answer. I have 2 ASAs forwarding their logs. I can search for one and find log data but the not the other one. I search thru metrics.log, license_usage.log, and splunkd.log. I find data about both of them in metrics and license, but only the one that works in splunkd.log. I find errors about events with no timestamp. And for both ASAs the license_usage.log file shows the same idx="xxxx" string.

So I am confused as to why I can't find any events for the second ASA.

[Richfez]

EdBruce,

While I applaud trying to search and find answers, you may be better served if you opened a new question and asked there, including all the specifics of your problem. I'm sure we can help you there!

This problem you posted the comment against already has two answers and there are many people who won't even bother looking at a problem with two answers, let alone adding a third answer.

[EdBruce]

Thanks for the reply and I will open a new question.

[wingfieldj]

The default search looks at logs posted to 'Main' Index. If you put the ASA logs into another index it does not show up. Changes to what index is used is under Access Control/Roles
change each account type, for example User
check under "Indexes searched by default"
add other logs as needed

[solarboyz1]

Assuming you are using a linux variant as your Splunk host, tcpdump shows that it is hitting the interface which means something higher up the stack is dropping them. If you have multiple interfaces on that server, you may want to look into reverse path filtering.