Why are Cisco ASA Logs are duplicated in SPLUNK?

[ornaldo]

Dear community,

After i forwarded the syslog from Cisco ASA into SPLUNK i noticed that the logs are duplicated and this is consuming our license. Any help please ? Thank You  

[ornaldo]

Thank You both @PickleRick @meetmshah 

 I tried to fw the logs of cisco into a syslog prior to splunk and the situation is the same.

So i will try to find if there is a bug with cisco asa

Br

[ornaldo]

Installed on the SPLUNK we do have Splunk Add-on for Cisco ASA. Maybe there is any misconfiguration here ?

[PickleRick]

Did you check any of the things I said or are you just shooting blindly?

So far you haven't provided us with much of a description of your architecture, you don't know your own ingestion process but you expect us to guess what's wrong with your setup.

[meetmshah]

AFAIK, Splunk Add-on for Cisco ASA inputs just monitors either syslog dumped files or monitors the port. In either of the case, Splunk will "ingest whatever is sent to it' and won't have anything to do with how events are being received or what happens before the file is written. So better way to troubleshoot is why the file itself has duplicate events (because Splunk only monitors what we ask for)

[PickleRick]

Events should not just duplicate on their own so if they do there must be a reason for this. It's hard to say without knowing your infrastructure.

Check your raw data on the wire (tcpdump/wireshark) if you're getting the events duplicated from the source. If not, check your configs further downstream.

[meetmshah]

Hello @ornaldo, per my experience, I feel the issue should be from Syslog end and not from Splunk. Can you please check if the events are duplicated from Syslog end? Maybe check in the raw file or create a tcpdump and validate the duplicate events?

[ornaldo]

Hi there,

I just checked the original logs stored on /opt/syslog/cisco_asa/XXXXXX and the log entries are doubled or even tripled.  Do you think it's a bug of cisco or any wrong configuration on the SPLUNK side ? I'm thinking most probably it's a SPLUNK misconfiguration since i do have 5 different cisco asa FWs and all have the same issue.

Thank You 

[PickleRick]

OK. So you're receiving your events from cisco via some third-party syslog daemon which writes them to a log file. From that log file you're ingesting the events to Splunk with a forwarder, right?

So if you have your events dobled or tripled in the file _before_ ingestion by Splunk it's clearly an issue "before" Splunk. So you have to check your syslog solution and your source.

[ornaldo]

The logs are being sent from CISCO ASA directly to SPLUNK without any syslog in the middle. 

[PickleRick · Accepted solution]

No, they are not. It might be the same host as your Splunk component (I don't know if it's a UF or if you're ingesting the data directly on your all-in-one server) but it most probably is a separate process which listens on a network socket and writes to the log files. (If splunk was receiving events directly on network port you wouldn't have any intermediate files on disk).

You can check with ss or netstat what is listening on yout 514 port (or any other port you're using for receiving syslog data). You could also do lsof or fuser to see which processes have those /opt/syslog files open apart from your splunk component.