Getting Data In

How to troubleshoot why I'm not getting syslog data from Cisco ASA into Splunk?

jimmycher
Engager

I have a Cisco ASA that is pushing out syslog files to the server that SPLUNK resides on. I verified they are reaching the server with TCPDump. The data is not getting into SPLUNK. Does the server need to be set up as a a syslog server, or does SPLUNK perform that function? What should be my troubleshooting steps?

0 Karma
1 Solution

Richfez
SplunkTrust
SplunkTrust

Splunk CAN ingest syslog inputs directly. You would create a data input listening on UDP (and possibly TCP) on port 514 for syslog. But just because you can doesn't mean you should.

A better way to do this would be to set up a syslog server (rsyslog or syslog-ng for *nix, or on windows I think the kiwi syslog daemon may still be free for this purpose?) and use that to gather the syslog inputs into files, then configure Splunk to read those files.

Why? How? Someone else has written up some great info on this, so check here. Give that a shot and let us know how it went!

View solution in original post

Richfez
SplunkTrust
SplunkTrust

Splunk CAN ingest syslog inputs directly. You would create a data input listening on UDP (and possibly TCP) on port 514 for syslog. But just because you can doesn't mean you should.

A better way to do this would be to set up a syslog server (rsyslog or syslog-ng for *nix, or on windows I think the kiwi syslog daemon may still be free for this purpose?) and use that to gather the syslog inputs into files, then configure Splunk to read those files.

Why? How? Someone else has written up some great info on this, so check here. Give that a shot and let us know how it went!

jimmycher
Engager

Thanks for prompt response Rich. I just found out at least one of my switches IoS files is being ingested directly into SPLUNK, and I haven't set that up as a syslog server yet. So your explanation matches my real-world, at least for IoS. Now I have to figure out why the ASA files are not being ingested. They have the same syslog host address, and both are being seen by the server (confirmed via TCPDump). Any thoughts?

0 Karma

Richfez
SplunkTrust
SplunkTrust

OK, a couple of things to check. In sort of vaguely some reasonable order.

Double-check that the ASA isn't syslogging to a different port. It could, for instance, be set up on 1514 or something else.

Then - how are you sure it's not coming in? Try a search like index=* host=X.X.X.X over all time and see if it shows up there. It's VERY possible it's just sent to a different index or set to an oddball sourcetype or something. In fact, you could even watch tcpdump, find an actual item that's reasonably unique that you can see come in, and search index=* mysearchstring to see if you can find that.

If you can find the events anywhere in Splunk, then look through the fields and see if you can figure out why you can't see it. We can help with that too.

Everything else I can think of that would make your system behave that way would be far less likely, but if one of those two isn't the problem we can help further, just tell us what you found.

0 Karma

jimmycher
Engager

I did as suggested, found unique items in TCPdump, but Splunk still does not see them. The TCP dump gives the source and destination IPs with syslog port (e.g. 10.10.10.1.syslog > 192.168.1.3.syslog). I think the port is correct. I left the "facility local 4 (20)", which is the default.

It's possible the defaults were tampered with on Splunk, I inherited this setup. What conf files should I double check?

0 Karma

Richfez
SplunkTrust
SplunkTrust

All right. So we know syslog packets are reaching the Splunk server from that host, and syslog packets are reaching the Splunk server from at least one other host. In the former case they're not "findable" but in the latter they are.

I expect then that there are entries we'll need to find. The following is in a few steps as we narrow down where to look.

First, run (Windows syntax first, then Linux syntax - you obviously only need one).

c:\program files\splunk\bin\splunk cmd btool --debug props list | findstr 10.10.10.1
/opt/splunk/bin/splunk cmd btool --debug props list | grep 10.10.10.1

(from now on, I'll use Windows syntax since I have a windows box open at the moment to copy/paste examples from. Adjust as required). You might get output like:

C:\Program Files\Splunk\etc\system\local\props.conf              [host::10.10.10.1]

If you do, GREAT! The file we need to check/change is $splunkhome/etc/system/local/props.conf.

If it doesn't return anything, then we'll look for the host that DOES work:

c:\program files\splunk\bin\splunk cmd btool --debug props list | findstr <host that works>

Then you'll know where that host is defined. If you edit the file it returns, you hopefully will see a few settings in there for that host. You'll want to copy/paste those to a new set, then change them as appropriate to make them apply to the other host as well.

Paste back what you've found if you can't figure it out from there.

Oh, one note: if any of the files are blah/blah/default/blah.conf, you should really REALLY make a copy of the pieces you need into blah/blah/local/blah.conf so they won't get overwritten at next upgrade.

0 Karma

jimmycher
Engager

I ran the commands, and got zero response, for both the switch and the Firewall. Yesterday, the switch syslog data was working fine, but today I only got one message on splunk, even though I've been changing all kinds of stuff on the switch. AGHHHH. The switch is still coming in as shown on TCPdump, but now even my reference point on SPLUNK is whacked.

As I mentioned, I have no idea what was customized on this app, and it is in a sandbox, so should my efforts now be redirected to a re-install? How do I do that and save the license info?

thanks.

0 Karma

Richfez
SplunkTrust
SplunkTrust

Well, we don't probably NEED to reinstall, but if you aren't very attached to the historical data you have, it's certainly a possibility.

First, though, I think you could try a simpler method.

Disable the input for UDP 514 in splunk.

Install syslog-ng (or rsyslog).

Configure syslog-ng (or rsyslog) to save incoming syslog to, say, /var/log/remote//log.txt. There are docs and info on why available. I think you'll need something like this in the syslog-ng config, though obviously sanity check it and make sure it's good for your environment.

source s_network_udp { udp( port(514)); };
source s_network_tcp { syslog( port(514) transport("tcp")); };
destination d_syslogs { file ("/var/log/remote/${HOST}/log.txt"); };
log {source(s_network_udp); source(s_network_tcp); destination(d_syslogs); };

That will create directories/files like /var/log/remote/10.10.10.1/log.txt. At this point, you've separated the syslog parts from the Splunk parts, so it makes life so much easier. Within moments you should see files being populated in those folders, check those and confirm they have the data you need. (Note, you MAY have to restart the logging on the cisco devices, though probably not).

Create new monitor inputs for those, using something like this as a guide:

[monitor:///var/log/remote/10.10.10.1/*.txt]
host_segment = 4
disabled = 0
index = firewall
sourcetype = cisco:asa

This will tell splunk to pick up the files from there, and use the 4th part of the path as the hostname to assign. NOTICE: change the index to whatever it is you need!

Then restart Splunk.

If you did it all, you'll now be able to search (in my example) index=firewall and see events from all hosts configured.

Obviously this is the short version. You may have to do a little playing, and I've tried to include docs or blog posts where applicable, but you still might have to do some digging. You could join up with the Splunk IRC channel, as long as you don't get too deep into wierd syslog problems, we can usually help there. 🙂

0 Karma

jcoates_splunk
Splunk Employee
Splunk Employee

port 514? That's below 1024, so you need to be root to open it. I'd use netstat -an to make sure it's really open. Or better yet, make your Splunk listen at 5514 and tell your Cisco to send there.

jackal713
Path Finder

Swicthing to a different port as jcoates suggests, worked for me.
I was having a similar issue. Syslog sending to Splunk was setup and working in Splunk. Data was being indexed and everything was working fine. Later, for reasons not related to Splunk, syslog sending was turned off, on the devices. After turning syslog sending back on, no data was being indexed. Switching to a different port for syslog made it work again.

P.S. I forgot to say THANK YOU!

0 Karma

Richfez
SplunkTrust
SplunkTrust

Nice catch, jcoates! If it's the original problem I'd be surprised since it was working for a box. BUT, it wouldn't be the first time I've seen something odd like that.

jimmycher: can you reconfigure your ASA to use port 5514 (or 5014, or 9514, or whatever > 1024 that's not already in use or a "regular" port lots of stuff uses - confirm as per jcoates' instructions), then tell Splunk to listen there?

0 Karma

jwelch_splunk
Splunk Employee
Splunk Employee

What are the results of:
1.
netstat -an |grep 514

2.
Who is splunk running as:
ps o ruser= -C splunkd

3.
your inputs related to udp:
/opt/splunk/bin/splunk btool inputs list --debug udp

4.
your inputs related to tcp:
/opt/splunk/bin/splunk btool inputs list --debug tcp

5.
iptables status:
iptables -L -vt nat

If you cannot reconfigure your ASA/Switches/Routers to hit a high port you can always use iptables to redirect to a high port if you are in fact running as a non-root user

Configure inputs.conf
[udp:9514]
[tcp:9514]

Put in iptables redirect rules:
iptables -t nat -A PREROUTING -p tcp -m tcp --dport 514 -j REDIRECT --to 9514
iptables -t nat -A PREROUTING -p udp -m udp --dport 514 -j REDIRECT --to 9514

Validate with iptables -L -vt nat

Data arriving on udp/tcp 514 will be sent to 9514

My stars are getting messed up for some reason replace shift8 with a real star
index=shift8 sourcetype=udpshift8 OR sourcetype=tcpshift8 alltime/realtime

Send yourself some test syslog messages to remove the asa from the picture:
nc -w0 -u 127.0.0.1 9514 <<< "Here with Test 1 via udp 9514"

index=* sourcetype=udp* OR sourcetype=tcp* alltime/realtime

And one last question does the box have multiple interfaces, if so could it be a reverse path filter issue?
results of:
cat /proc/sys/net/ipv4/conf/default/rp_filter

jwelch_splunk
Splunk Employee
Splunk Employee

One other question:

Results of:

netstat -su

0 Karma

jwelch_splunk
Splunk Employee
Splunk Employee

what happens if you do:

no logging host inside xxx.xxx.xxx.xxx for example.

After you issue that do you see the the traffic stop on the tcpdump?

now do a:

logging host inside xxx.xxx.xxx.xxx

Do you see the traffic on the tcpdump again? If so do you now see the traffic in splunk?

I have seen the data arrive on a dump before but syslog on the foriegn system had to be restarted on an ASA it would be the no logging host command as an example.

jimmycher
Engager

Yes, the tcpdump will stop incrementing when I remove the logging host command on my switch. When I reapply, the tcpdump then shows the syslog files. But SPLUNK does not show it at all, except for one instance two hours ago (???). So now I can no longer get switch syslog in, and still can't get ASA syslog data in. Maybe time for a reinstall?

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...