Getting Data In
Highlighted

How to troubleshoot why I'm not getting syslog data from Cisco ASA into Splunk?

Engager

I have a Cisco ASA that is pushing out syslog files to the server that SPLUNK resides on. I verified they are reaching the server with TCPDump. The data is not getting into SPLUNK. Does the server need to be set up as a a syslog server, or does SPLUNK perform that function? What should be my troubleshooting steps?

0 Karma
Highlighted

Re: How to troubleshoot why I'm not getting syslog data from Cisco ASA into Splunk?

SplunkTrust
SplunkTrust

Splunk CAN ingest syslog inputs directly. You would create a data input listening on UDP (and possibly TCP) on port 514 for syslog. But just because you can doesn't mean you should.

A better way to do this would be to set up a syslog server (rsyslog or syslog-ng for *nix, or on windows I think the kiwi syslog daemon may still be free for this purpose?) and use that to gather the syslog inputs into files, then configure Splunk to read those files.

Why? How? Someone else has written up some great info on this, so check here. Give that a shot and let us know how it went!

View solution in original post

Highlighted

Re: How to troubleshoot why I'm not getting syslog data from Cisco ASA into Splunk?

Engager

Thanks for prompt response Rich. I just found out at least one of my switches IoS files is being ingested directly into SPLUNK, and I haven't set that up as a syslog server yet. So your explanation matches my real-world, at least for IoS. Now I have to figure out why the ASA files are not being ingested. They have the same syslog host address, and both are being seen by the server (confirmed via TCPDump). Any thoughts?

0 Karma
Highlighted

Re: How to troubleshoot why I'm not getting syslog data from Cisco ASA into Splunk?

SplunkTrust
SplunkTrust

OK, a couple of things to check. In sort of vaguely some reasonable order.

Double-check that the ASA isn't syslogging to a different port. It could, for instance, be set up on 1514 or something else.

Then - how are you sure it's not coming in? Try a search like index=* host=X.X.X.X over all time and see if it shows up there. It's VERY possible it's just sent to a different index or set to an oddball sourcetype or something. In fact, you could even watch tcpdump, find an actual item that's reasonably unique that you can see come in, and search index=* mysearchstring to see if you can find that.

If you can find the events anywhere in Splunk, then look through the fields and see if you can figure out why you can't see it. We can help with that too.

Everything else I can think of that would make your system behave that way would be far less likely, but if one of those two isn't the problem we can help further, just tell us what you found.

0 Karma
Highlighted

Re: How to troubleshoot why I'm not getting syslog data from Cisco ASA into Splunk?

Engager

I did as suggested, found unique items in TCPdump, but Splunk still does not see them. The TCP dump gives the source and destination IPs with syslog port (e.g. 10.10.10.1.syslog > 192.168.1.3.syslog). I think the port is correct. I left the "facility local 4 (20)", which is the default.

It's possible the defaults were tampered with on Splunk, I inherited this setup. What conf files should I double check?

0 Karma
Highlighted

Re: How to troubleshoot why I'm not getting syslog data from Cisco ASA into Splunk?

Splunk Employee
Splunk Employee

what happens if you do:

no logging host inside xxx.xxx.xxx.xxx for example.

After you issue that do you see the the traffic stop on the tcpdump?

now do a:

logging host inside xxx.xxx.xxx.xxx

Do you see the traffic on the tcpdump again? If so do you now see the traffic in splunk?

I have seen the data arrive on a dump before but syslog on the foriegn system had to be restarted on an ASA it would be the no logging host command as an example.

Highlighted

Re: How to troubleshoot why I'm not getting syslog data from Cisco ASA into Splunk?

Engager

Yes, the tcpdump will stop incrementing when I remove the logging host command on my switch. When I reapply, the tcpdump then shows the syslog files. But SPLUNK does not show it at all, except for one instance two hours ago (???). So now I can no longer get switch syslog in, and still can't get ASA syslog data in. Maybe time for a reinstall?

0 Karma
Highlighted

Re: How to troubleshoot why I'm not getting syslog data from Cisco ASA into Splunk?

SplunkTrust
SplunkTrust

All right. So we know syslog packets are reaching the Splunk server from that host, and syslog packets are reaching the Splunk server from at least one other host. In the former case they're not "findable" but in the latter they are.

I expect then that there are entries we'll need to find. The following is in a few steps as we narrow down where to look.

First, run (Windows syntax first, then Linux syntax - you obviously only need one).

c:\program files\splunk\bin\splunk cmd btool --debug props list | findstr 10.10.10.1
/opt/splunk/bin/splunk cmd btool --debug props list | grep 10.10.10.1

(from now on, I'll use Windows syntax since I have a windows box open at the moment to copy/paste examples from. Adjust as required). You might get output like:

C:\Program Files\Splunk\etc\system\local\props.conf              [host::10.10.10.1]

If you do, GREAT! The file we need to check/change is $splunkhome/etc/system/local/props.conf.

If it doesn't return anything, then we'll look for the host that DOES work:

c:\program files\splunk\bin\splunk cmd btool --debug props list | findstr <host that works>

Then you'll know where that host is defined. If you edit the file it returns, you hopefully will see a few settings in there for that host. You'll want to copy/paste those to a new set, then change them as appropriate to make them apply to the other host as well.

Paste back what you've found if you can't figure it out from there.

Oh, one note: if any of the files are blah/blah/default/blah.conf, you should really REALLY make a copy of the pieces you need into blah/blah/local/blah.conf so they won't get overwritten at next upgrade.

0 Karma
Highlighted

Re: How to troubleshoot why I'm not getting syslog data from Cisco ASA into Splunk?

Engager

I ran the commands, and got zero response, for both the switch and the Firewall. Yesterday, the switch syslog data was working fine, but today I only got one message on splunk, even though I've been changing all kinds of stuff on the switch. AGHHHH. The switch is still coming in as shown on TCPdump, but now even my reference point on SPLUNK is whacked.

As I mentioned, I have no idea what was customized on this app, and it is in a sandbox, so should my efforts now be redirected to a re-install? How do I do that and save the license info?

thanks.

0 Karma
Highlighted

Re: How to troubleshoot why I'm not getting syslog data from Cisco ASA into Splunk?

SplunkTrust
SplunkTrust

Well, we don't probably NEED to reinstall, but if you aren't very attached to the historical data you have, it's certainly a possibility.

First, though, I think you could try a simpler method.

Disable the input for UDP 514 in splunk.

Install syslog-ng (or rsyslog).

Configure syslog-ng (or rsyslog) to save incoming syslog to, say, /var/log/remote//log.txt. There are docs and info on why available. I think you'll need something like this in the syslog-ng config, though obviously sanity check it and make sure it's good for your environment.

source s_network_udp { udp( port(514)); };
source s_network_tcp { syslog( port(514) transport("tcp")); };
destination d_syslogs { file ("/var/log/remote/${HOST}/log.txt"); };
log {source(s_network_udp); source(s_network_tcp); destination(d_syslogs); };

That will create directories/files like /var/log/remote/10.10.10.1/log.txt. At this point, you've separated the syslog parts from the Splunk parts, so it makes life so much easier. Within moments you should see files being populated in those folders, check those and confirm they have the data you need. (Note, you MAY have to restart the logging on the cisco devices, though probably not).

Create new monitor inputs for those, using something like this as a guide:

[monitor:///var/log/remote/10.10.10.1/*.txt]
host_segment = 4
disabled = 0
index = firewall
sourcetype = cisco:asa

This will tell splunk to pick up the files from there, and use the 4th part of the path as the hostname to assign. NOTICE: change the index to whatever it is you need!

Then restart Splunk.

If you did it all, you'll now be able to search (in my example) index=firewall and see events from all hosts configured.

Obviously this is the short version. You may have to do a little playing, and I've tried to include docs or blog posts where applicable, but you still might have to do some digging. You could join up with the Splunk IRC channel, as long as you don't get too deep into wierd syslog problems, we can usually help there. 🙂

0 Karma