How to send Cisco ASA Firewall logs to syslog-ng server and push it out to the indexer with universal forwarder

[jhl226116]

Hi Guys,

I've been trying to send Cisco ASA firewall logs to syslog-ng server where the forwarder is installed but I just can't get it working.

I've setup a forwarder and installed syslog-ng in Ubuntu VM.

I have tried to follow the instructions on this link and also from other various sources but I'm stressful enough to say that I just can't get it working.

https://www.splunk.com/blog/2016/03/11/using-syslog-ng-with-splunk/

I really need some clear detailed step by step instructions on how to configure Cisco ASA to store syslogs into the syslog-ng server and forward the data to indexer.

I'm not sure if I configured syslog-ng server properly in Ubuntu. I used following tutorial but no success.

https://www.youtube.com/watch?v=glvsJJFbzZc&feature=em-share_video_user

Could you check if everything looks ok below and advise the next step from here?

My goal is to send Cisco ASA Firewall logs to syslog-ng server and push it out to the indexer with universal forwarder so that I'm able to see all the cisco asa logs from the search.

My setup is as below: All servers have been built with Ubuntu in VM.

Indexer: 10.10.50.11
Forwarder: 10.10.50.12 (Installed syslog-ng here)

I can ping and SSH between Indexer and forwarder.

Configured universal forwarder to send data to the receiving indexer.
root@forwarder:/opt/splunkforwarder/bin# ./splunk add forward-server 10.10.50.11:9997 -auth admin:seeshock
 root@forwarder:/opt/splunkforwarder/bin# ./splunk list forward-server
Active forwards:
None
Configured but inactive forwards:
10.10.50.11:9997

Configure deployment client
root@forwarder:/opt/splunkforwarder/bin# ./splunk show deploy-poll
Deployment Server URI is set to "10.10.50.12:8089".

root@indexer:/opt/splunk/bin# ./splunk status
splunkd is running (PID: 3109).
splunk helpers are running (PIDs: 3110 3118 3183 3200).
 

root@indexer:/opt/splunk/bin# ./splunk btool inputs list splunktcp --debug | grep -v default
/opt/splunk/etc/system/local/inputs.conf host = indexer
/opt/splunk/etc/apps/launcher/local/inputs.conf [splunktcp://9997]
/opt/splunk/etc/apps/launcher/local/inputs.conf connection_host = ip
/opt/splunk/etc/system/local/inputs.conf host = indexer
 

root@indexer:/opt/splunk/bin# ./splunk list inputstatus
tcp_cooked:listenerports :
9997
 
 
 
 
index=internal host="indexer"
 
Time Event

4/2/17 10.10.50.11 - admin [02/Apr/2017:20:47:25.825 +1000] "GET /en-US/splunkd/raw/services/search/shelper?output_mode=json&snippet=true&snippetEmbedJS=false&namespace=search&search=search+index%3D_internal+host%3D&useTypeahead=true&useAssistant=false&showCommandHelp=true&showCommandHistory=true&showFieldInfo=false&=1491126297345 HTTP/1.1" 200 5141 "http://10.10.50.11:8000/en-US/app/search/search?q=search%20index%3D_internal%20host%3Dforwarder&disp..." "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:51.0) Gecko/20100101 Firefox/51.0" - b44fb5fd27b3677a2443d6de473370fe 8ms
8:47:25.825 PM · host = indexer
· source = /opt/splunk/var/log/splunk/splunkd_ui_access.log
· sourcetype = splunkd_ui_access
4/2/17 10.10.50.11 - admin [02/Apr/2017:20:47:25.681 +1000] "GET /en-US/splunkd/_raw/services/search/shelper?output_mode=json&snippet=true&snippetEmbedJS=false&namespace=search&search=search+index%3D_internal+host%3Df&useTypeahead=true&useAssistant=false&showCommandHelp=true&showCommandHistory=true&showFieldInfo=false&=1491126297344 HTTP/1.1" 200 5109 "http://10.10.50.11:8000/en-US/app/search/search?q=search%20index%3D_internal%20host%3Dforwarder&disp..." "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:51.0) Gecko/20100101 Firefox/51.0" - b44fb5fd27b3677a2443d6de473370fe 40ms
8:47:25.681 PM · host = indexer
· source = /opt/splunk/var/log/splunk/splunkd_ui_access.log
· sourcetype = splunkd_ui_access

index=_internal host="forwarder"
 
4/3/17 04-03-2017 14:15:24.393 +1000 INFO DC:PhonehomeThread - Attempted handshake 260 times. Will try to re-subscribe to handshake reply

2:15:24.393 PM · host = forwarder
· source = /opt/splunkforwarder/var/log/splunk/splunkd.log
· sourcetype = splunkd
4/3/17 04-03-2017 14:15:24.393 +1000 INFO DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
2:15:24.393 PM · host = forwarder
· source = /opt/splunkforwarder/var/log/splunk/splunkd.log
· sourcetype = splunkd
4/3/17 04-03-2017 14:15:16.083 +1000 INFO Metrics - group=thruput, name=uncooked_output, instantaneous_kbps=0.000000, instantaneous_eps=0.000000, average_kbps=0.000000, total_k_processed=0.000000, kb=0.000000, ev=0.000000
2:15:16.083 PM · host = forwarder
· source = /opt/splunkforwarder/var/log/splunk/metrics.log
· sourcetype = splunkd

 
 
root@indexer:/etc/syslog-ng# netstat -an | grep 9997
tcp 0 0 0.0.0.0:9997 0.0.0.0:* LISTEN
tcp 0 0 10.10.50.11:9997 10.10.50.12:53380 ESTABLISHED
 
 

 
root@indexer:/opt/splunk/etc/system/local# mkdir firewall_asa

root@indexer:/opt/splunk/etc/system/local# cd firewall_asa

root@indexer:/opt/splunk/etc/system/local/firewall_asa# cd

root@indexer:~# cd /opt/splunk/bin

root@indexer:/opt/splunk/bin# ./splunk add udp 514 -sourcetype cisco:asa
Listening for UDP input on port 514.
 

root@forwarder:/opt/splunkforwarder/bin# ./splunk show deploy-poll
Deployment Server URI is set to "10.10.50.12:8089".

root@forwarder:/opt/splunkforwarder/bin# ./splunk list forward-server
Active forwards:
10.10.50.11:9997
Configured but inactive forwards:
None

root@forwarder:/opt/splunkforwarder/bin# ./splunk show servername
Server name: forwarder

root@forwarder:/opt/splunkforwarder/bin# ./splunk show default-hostname
Default hostname for data inputs: forwarder.

root@forwarder:/opt/splunkforwarder/bin# ./splunk show servername
Server name: forwarder

root@forwarder:/opt/splunkforwarder/bin# ./splunk show default-hostname
Default hostname for data inputs: forwarder.

root@forwarder:/opt/splunkforwarder/bin# ./splunk add forward-server 10.10.50.11:9997 -auth admin:seeshock
Added forwarding to: 10.10.50.11:9997.
 
root@forwarder:/opt/splunkforwarder/bin# ./splunk list forward-server
Active forwards:
None
Configured but inactive forwards:
10.10.50.11:9997
 
 
 

[jhl226116]

Hi Guys,

Think I got it working!

I ran a search sourcetype="cisco:asa" and it brought up 484,558 events.

Now I have tons of new questions with things like how to filter syslogs and how to setup and configure NTP as my time stamps are wrong and alot more but for now I'm so happy I got it to work finally!!

Thanks to all of you guys who helped me to achieve this I couldn't have done it if I hadn't come to this forum with absolutely no help from anywhere else.

One more thing I'd like to ask is with the query commands, could someone tell me how I can do standard, general and more discoverable searches with filters, limitations etc with Cisco asa?

All the search command I know is sourcetype="cisco:asa"

Thanks heaps guys, really appreciate it.

Now I'll see if I can get the logs in for the routers and switches as well.

[richgalloway]

If your problem is resolved, please accept one of the answers (perhaps your own) to help future searchers.

[jhl226116]

Done, I couldn't accept my own but all good. I also didn't know that, thanks for the tip.

[nychawk]

You can narrow down your search by selecting any of the fields on the left column, then selecting any of these, i.e. src_ip, etc.

Look at the Cisco ASA app, has some pretty good searches built-in.

[jhl226116]

Thanks, I've installed Cisco ASA app, will look into that. It's not showing vpn geolocation etc but will try to dig into that as well.

[alemarzu · Accepted solution]

Hi there,

Am I wrong or you just create the UDP input on your indexer rather than the Forwarder ?

"root@indexer:/opt/splunk/bin# ./splunk add udp 514 -sourcetype cisco:asa
Listening for UDP input on port 514."

This are the basic steps to do what you need, assuming that you have firewall rules/port opened already.

1. Enable listen on port 9997 on the Indexer to recieve incoming data from the UF.

   ./splunk enable listen 9997 -auth username:password
   

2. Configure your forwarder to send data to the indexer.

   ./splunk add forward-server <IP OR DNS>:9997
   

3. Create the UDP input on your UF to listen that syslog.

   ./splunk add udp 514 -sourcetype cisco:asa
   

   Hope it helps.

[jhl226116]

You're right, I created UDP input in the indexer rather than in the forwarder. Maybe that's why something's been screwing up. I didn't know that.
I've now configured UDP 514 input in the forwarder where the syslog-ng server is installed and removed it from the indexer.
Port 9997 was already enabled in the indexer and add forward-server is already present as well since I've configured it before.
See the output below:

1. Enable listen on port 9997 on the Indexer to recieve incoming data from the UF. <<<<< This has been done ./splunk enable listen 9997 -auth username:password

root@indexer:/opt/splunk/bin# ./splunk enable listen 9997 -auth admin:seeshock
Failed to create. Configuration for port 9997 already exists.

1. Configure your forwarder to send data to the indexer. <<<<<<<<<<<< This has been done ./splunk add forward-server :9997

root@forwarder:/opt/splunkforwarder/bin# ./splunk add forward-server 10.10.50.11:9997
10.10.50.11:9997 forwarded-server already present

1. Create the UDP input on your UF to listen that syslog. <<<<<<<<<<<<<<< This has been done ./splunk add udp 514 -sourcetype cisco:asa

root@forwarder:/opt/splunkforwarder/bin# ./splunk add udp 514 -sourcetype cisco:asa
Listening for UDP input on port 514.

Also Removed UDP 514 from indexer

root@indexer:/opt/splunk/bin# ./splunk remove udp 514 -sourcetype cisco:asa
Removed udp://514

Now, how do I test everything is working properly? Though, I'm still not confident syslog-ng server is configured properly with Cisco ASA. I need some clear instructions on this. Any help would be appreciated.

[nychawk]

For starters, you cannot run syslog-NG and have your UF listening on udp 514, that will cause a conflict.

1. Either stop//disable syslog-NG, or stop the UF from listening on that port; your choice, but not both.
2. On your inputs on your UF, you will need to point to the file that syslog-NG creates and fills, or, if you chose to listen to udp/514 on the UF, then do nothing.

I see you have either given up, or about to give up, on syslog-NG, not a good choice. I offered to review your syslog-ng.conf, but you never posted that.

[jhl226116]

oh when did you ask me to post syslog-ng.conf? I wasn't sure how to do that. I see it's in the last line.
Let me do that for you, just trying to figure out how.

[nychawk]

You stated you setup syslog-ng on your UF, the .conf is usually in either /etc or /use/local/etc, run a find

run this:
netstat -tulpn

The above will show ports open, and by which process. Need to determine if udp 514 is open by Splunk or syslogng

[jhl226116]

"You stated you setup syslog-ng on your UF, the .conf is usually in either /etc or /use/local/etc, run a find"

Are you referring to inputs.conf and outputs.conf?

[jhl226116]

This indicates udp514 is open by Splunkd isn't it?

root@forwarder:~# netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.1.1:53 0.0.0.0: LISTEN 799/dnsmasq
tcp 0 0 0.0.0.0:22 0.0.0.0: LISTEN 755/sshd
tcp 0 0 0.0.0.0:8089 0.0.0.0: LISTEN 1338/splunkd
tcp6 0 0 :::22 ::: LISTEN 755/sshd
udp 0 0 0.0.0.0:5353 0.0.0.0: 544/avahi-daemon: r
udp 0 0 0.0.0.0:54600 0.0.0.0: 544/avahi-daemon: r
udp 0 0 127.0.1.1:53 0.0.0.0: 799/dnsmasq
udp 0 0 0.0.0.0:514 0.0.0.0: 1338/splunkd <<<<<<<<<
udp 0 0 0.0.0.0:631 0.0.0.0: 630/cups-browsed
udp6 0 0 :::5353 ::: 544/avahi-daemon: r
udp6 0 0 :::44497 :::* 544/avahi-daemon: r
root@forwarder:~#

[alemarzu]

Hello again,

To test if splunk is working correctly, you can deploy a random input to the UF.

To test if syslog is configured properly, you can configure your syslog client (UF) to write the incoming data to a file.

[jhl226116]

Can you give me an example?

When I goto Add data>Forward it says, there are currently no forwarders configured as deployment clients to this instance.

I'm not sure if that's normal.

Can you do it from the terminal CLI?

[alemarzu]

Run this lines on your forwarder.

• Configure UF as a client: ./splunk set deploy-poll <indexer_ip/dns>:8089
• To create a quick random input to test your configuration just execute this on the UF: ./splunk add monitor /opt/splunk/copyright.txt -sourcetype configtest -index main

Search for index=main sourcetype=configtest on your indexer to see if your configuration is working.

[jhl226116]

root@forwarder:/opt/splunkforwarder/bin# ./splunk set deploy-poll 10.10.50.11:8089
root@forwarder:/opt/splunkforwarder/bin# ./splunk add monitor /opt/splunk/copyright.txt -sourcetype configtest -index main
Parameter name: Path does not exist.

root@forwarder:/opt/splunkforwarder/bin# ./splunk add monitor /opt/splunkforwarder/copyright.txt -sourcetype configtest -index main
Added monitor of '/opt/splunkforwarder/copyright.txt'.

In the forwarder I configured add monitor but the Path does not exist on /opt/splunk/copyright.txt so I added on /opt/splunkforwarder/copyright.txt

However, when I do a search index=main sourcetype="configtest"  there were No results found.

[nychawk]

Try this:

1. Run traceroute on your ubu box to first determine if you are getting data from your ASA:

tcpdump -vvv -i src host and udp port 514
or
tcpdump -vvv -i src host and tcp port 514

Assuming you see data, then move on to the next steps, else you are either not sending logs from your ASA, you have a routing issue (which I doubt as you are able to ping your syslog server), or are perhaps running SELinux and are blocked?

If you see data, then your issues are with either your syslog-NG daemon, or its config, most likely its config.

You need to have syslog listening on the appropriate protocol, dependent on whether you are sending to tcp or udp; you can listen on both for syslog-NG. Also, aside from a listener, you must objects such as source, destination, log, filter, parser, rewrite rule, or template.

If you see data in tcpdump, then copy/paste your syslog-NG config file into this issue, please remove your comments.

[jhl226116]

Cisco ASA: 10.10.50.1
Indexer: 10.10.50.11
Forwarder: 10.10.50.12

In the command line, are you referring "src host" as indexer or Cisco asa?
I tried to run the command from forwarder pointing src host as both indexer and cisco asa but result shows No such device. Am I running the command correctly and from the right directory location?
Sorry I'm a newbie just 4mth into Splunk, trying to understand how everything works. It's quite difficult trying to learn everything by myself alone, it would be x10 times faster if someone taught me how to do it.

Please see output below:

From Forwarder pointing src host as Indexer 10.10.50.11
root@forwarder:~# tcpdump -vvv -i 10.10.50.11 udp port 514
tcpdump: 10.10.50.11: SIOCETHTOOL(ETHTOOL_GET_TS_INFO) ioctl failed: No such device
 
root@forwarder:~# tcpdump -vvv -i 10.10.50.11 tcp port 514
tcpdump: 10.10.50.11: SIOCETHTOOL(ETHTOOL_GET_TS_INFO) ioctl failed: No such device

From Forwarder pointing src host as Cisco ASA 10.10.50.1
root@forwarder:~# tcpdump -vvv -i 10.10.50.1 udp  port 514
tcpdump: 10.10.50.1: SIOCETHTOOL(ETHTOOL_GET_TS_INFO) ioctl failed: No such device

root@forwarder:~# tcpdump -vvv -i 10.10.50.1 tcp  port 514
tcpdump: 10.10.50.1: SIOCETHTOOL(ETHTOOL_GET_TS_INFO) ioctl failed: No such device

I'm not sure what's going on but my gut tells me that Ciasco asa is not communicating with syslog-ng which is installed on the forwarder server.

I installed TCPdump and tried running it for the first time use, I'm not sure if some ports are missing or anything but I'm not sure where to look.
Does everything look ok? Or anything doesn't look right?

TCPDUMP output from forwarder

root@forwarder:~# tcpdump -i ens160
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens160, link-type EN10MB (Ethernet), capture size 262144 bytes
10:08:21.205181 IP 10.10.50.1.syslog > 10.10.50.12.syslog: SYSLOG local5.error, length: 141
10:08:21.205221 IP 10.10.50.12 > 10.10.50.1: ICMP 10.10.50.12 udp port syslog unreachable, length 177
10:08:21.205963 IP 10.10.50.12.28344 > dns1.tpgi.com.au.domain: 38283+ PTR? 12.50.10.10.in-addr.arpa. (42)
10:08:21.208036 IP dns1.tpgi.com.au.domain > 10.10.50.12.28344: 38283 NXDomain 0/1/0 (119)
10:08:21.208235 IP 10.10.50.12.45615 > dns1.tpgi.com.au.domain: 57567+ PTR? 1.50.10.10.in-addr.arpa. (41)
10:08:21.209522 IP dns1.tpgi.com.au.domain > 10.10.50.12.45615: 57567 NXDomain 0/1/0 (118)
10:08:21.209752 IP 10.10.50.12.38074 > dns1.tpgi.com.au.domain: 11768+ PTR? 35.160.12.203.in-addr.arpa. (44)
10:08:21.211518 IP dns1.tpgi.com.au.domain > 10.10.50.12.38074: 11768 1/2/0 PTR dns1.tpgi.com.au. (110)
10:08:21.332932 IP 10.10.50.12.35998 > 10.10.50.11.9997: Flags [P.], seq 4241108933:4241109329, ack 1934590004, win 237, options [nop,nop,TS val 128417542 ecr 285094319], length 396
10:08:21.333142 IP 10.10.50.11.9997 > 10.10.50.12.35998: Flags [.], ack 396, win 1452, options [nop,nop,TS val 285094735 ecr 128417542], length 0
[2]+ Stopped tcpdump -i ens160

TCPDUMP output from Indexer

root@indexer:~# tcpdump -i ens160
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens160, link-type EN10MB (Ethernet), capture size 262144 bytes
17:35:54.469974 IP 10.10.50.12.35998 > 10.10.50.11.9997: Flags [P.], seq 4241118889:4241119208, ack 1934590004, win 237, options [nop,nop,TS val 128428299 ecr 285104916], length 319
17:35:54.470013 IP 10.10.50.12.60808 > 10.10.50.11.9997: Flags [S], seq 604967860, win 29200, options [mss 1460,sackOK,TS val 128428299 ecr 0,nop,wscale 7], length 0
17:35:54.470038 IP 10.10.50.11.9997 > 10.10.50.12.35998: Flags [.], ack 319, win 1452, options [nop,nop,TS val 285105492 ecr 128428299], length 0
17:35:54.470052 IP 10.10.50.11.9997 > 10.10.50.12.60808: Flags [S.], seq 2536124428, ack 604967861, win 28960, options [mss 1460,sackOK,TS val 285105492 ecr 128428299,nop,wscale 7], length 0
17:35:54.470199 IP 10.10.50.12.60808 > 10.10.50.11.9997: Flags [.], ack 1, win 229, options [nop,nop,TS val 128428299 ecr 285105492], length 0
17:35:54.470254 IP 10.10.50.12.60808 > 10.10.50.11.9997: Flags [F.], seq 1, ack 1, win 229, options [nop,nop,TS val 128428299 ecr 285105492], length 0
17:35:54.470520 IP 10.10.50.11.9997 > 10.10.50.12.60808: Flags [F.], seq 1, ack 2, win 227, options [nop,nop,TS val 285105492 ecr 128428299], length 0
17:35:59.441948 IP 10.10.50.12.35998 > 10.10.50.11.9997: Flags [P.], seq 319:688, ack 1, win 237, options [nop,nop,TS val 128429542 ecr 285105492], length 369
17:35:59.442022 IP 10.10.50.11.9997 > 10.10.50.12.35998: Flags [.], ack 688, win 1452, options [nop,nop,TS val 285106735 ecr 128429542], length 0
17:35:59.442137 IP 10.10.50.12.35998 > 10.10.50.11.9997: Flags [P.], seq 688:779, ack 1, win 237, options [nop,nop,TS val 128429542 ecr 285106735], length 91
17:35:59.442169 IP 10.10.50.11.9997 > 10.10.50.12.35998: Flags [.], ack 779, win 1452, options [nop,nop,TS val 285106735 ecr 128429542], length 0

[7]+ Stopped tcpdump -i ens160

[nychawk]

This site totally mangled my suggestion, but now that I know your interface name, it is this from your syslog-NG server:

tcpdump -vvv -i ens160 udp port 514

If you see data coming from your ASA firewall, then your problem is with your syslog=NG config, which I've asked you to post, but you did not. As I've said before, I suspect your issue is most likely with your syslog server, given setting up an ASA to send to syslog is pretty straight-forward. Another root cause might be how you've setup or used DNS, see last line in his comment.

Based on your output, I am guessing the following:

indexer is 10.10.50.11
your universal forwarder/syslog server is 10.10.50.11
ASA Firewall is 10.10.50.1

Correct?

Did you configure each of these to communicate using hostnames, or IP addresses? I ask because your tcpdump is showing attempts to resolve internal IP addresses, which will not, should not work.

1. Check that you ASA is sending to an IP address, not a hostname.
2. Check that your syslog-NG server, which is running your UF, is sending (output.conf) to your indexer using an IP address, not a hostname.

[jhl226116]

Sorry for the delayed response, I was on a 5 day Easter long weekend break and just got back to work. Thanks for your help.
I ran your command and it seems like I'm getting something from the ASA firewall. Do you see the output as data coming in from ASA firewall? If that's the case then as you already predicted it could be a misconfigured syslog-ng which I agree to as I was never confident with it because none of the
Instructions really worked well and I don't know what I did really. I need some clear instructions on this.

root@forwarder:~# tcpdump -vvv -i ens160 udp port 514
tcpdump: listening on ens160, link-type EN10MB (Ethernet), capture size 262144 bytes
10:01:14.409934 IP (tos 0x0, ttl 255, id 4312, offset 0, flags [none], proto UDP (17), length 188)
10.10.50.1.syslog > 10.10.50.12.syslog: [udp sum ok] SYSLOG, length: 160
Facility local5 (21), Severity critical (2)
Msg: Apr 19 2017 11:05:08 CSOCLABFW01 : %ASA-2-106001: Inbound TCP connection denied from 178.47.249.32/22733 to 61.68.11.85/23 flags SYN on interface outside\0x0a
0x0000: 3c31 3730 3e41 7072 2031 3920 3230 3137
0x0010: 2031 313a 3035 3a30 3820 4353 4f43 4c41
0x0020: 4246 5730 3120 3a20 2541 5341 2d32 2d31
0x0030: 3036 3030 313a 2049 6e62 6f75 6e64 2054
0x0040: 4350 2063 6f6e 6e65 6374 696f 6e20 6465
0x0050: 6e69 6564 2066 726f 6d20 3137 382e 3437
0x0060: 2e32 3439 2e33 322f 3232 3733 3320 746f
0x0070: 2036 312e 3638 2e31 312e 3835 2f32 3320
0x0080: 666c 6167 7320 5359 4e20 206f 6e20 696e
0x0090: 7465 7266 6163 6520 6f75 7473 6964 650a
10:01:19.657510 IP (tos 0x0, ttl 255, id 6363, offset 0, flags [none], proto UDP (17), length 197)
10.10.50.1.syslog > 10.10.50.12.syslog: [udp sum ok] SYSLOG, length: 169
Facility local5 (21), Severity warning (4)
Msg: Apr 19 2017 11:05:14 CSOCLABFW01 : %ASA-4-106023: Deny udp src inside:10.10.50.2/1028 dst outside:172.30.234.150/5516 by access-group "inside_access_in" [0x0, 0x0]\0x0a
0x0000: 3c31 3732 3e41 7072 2031 3920 3230 3137
0x0010: 2031 313a 3035 3a31 3420 4353 4f43 4c41
0x0020: 4246 5730 3120 3a20 2541 5341 2d34 2d31
0x0030: 3036 3032 333a 2044 656e 7920 7564 7020
0x0040: 7372 6320 696e 7369 6465 3a31 302e 3130
0x0050: 2e35 302e 322f 3130 3238 2064 7374 206f
0x0060: 7574 7369 6465 3a31 3732 2e33 302e 3233
0x0070: 342e 3135 302f 3535 3136 2062 7920 6163
0x0080: 6365 7373 2d67 726f 7570 2022 696e 7369
0x0090: 6465 5f61 6363 6573 735f 696e 2220 5b30
0x00a0: 7830 2c20 3078 305d 0a
10:01:19.657586 IP (tos 0x0, ttl 255, id 25234, offset 0, flags [none], proto UDP (17), length 197)
10.10.50.1.syslog > 10.10.50.12.syslog: [udp sum ok] SYSLOG, length: 169

root@forwarder:~# tcpdump -vvv -i ens160 tcp port 514
tcpdump: listening on ens160, link-type EN10MB (Ethernet), capture size 262144 bytes

I've already provided IP's for the ASA, Indexer and Forwarder in my previous post (In the very beginning).

Cisco ASA: 10.10.50.1
Indexer: 10.10.50.11
Forwarder: 10.10.50.12

I think I configured each of these to communicate using IP addresses, not the hostnames but how do I check?

1. Check that you ASA is sending to an IP address, not a hostname.
   I think it's using an IP address, not a hostname but how do I check?

2. Check that your syslog-NG server, which is running your UF, is sending (output.conf) to your indexer using an IP address, not a hostname.
   I think it's using an IP address, not a hostname but how do I check?
   root@forwarder:~# nano /opt/splunkforwarder/etc/system/local/outputs.conf
   [tcpout]
   defaultGroup = default-autolb-group
   [tcpout:default-autolb-group]
   server = 10.10.50.11:9997
   [tcpout-server://10.10.50.11:9997]