Splunk Running with Firewall/iptables Enabled

[eandresen]

I am experiencing some strange issues with Splunk running properly and be responsive while the Linux Firewall/IPTables are enabled.

When the iptables service is running, Splunk is unresponsive to CLI commands through SSH and the webpage throws a "503 - Service Unavailable" following an attempted login with the admin account. I soon as I stop the iptables services, everything works fine. I can issue commands like "./splunk show splunkd-port" and "./splunk show web-port" without problems and logging into the webpage works great.

Below are the basics of my setup:

• Running CentOS 6.3 (fully updated) on a VMWare server.
• Running Splunk 5.0.1 version.

Below are the Splunk ports I am using:

• Web port: 443 (SSL enabled)
• Splunkd port: 8089
• Splunk Forwarder: 9997

Below are the settings within the "/etc/sysconfig/iptables" file:

# Firewall configuration written by system-config-firewall
# Manual customization of this file is not recommended.
*filter :INPUT DROP [0:0] :OUTPUT DROP [0:0] :FORWARD DROP [0:0]

# Open port for ping
-A INPUT -p icmp -j ACCEPT

# Open port for SSH
-A INPUT -i eth0 -p tcp --dport 22 -m state --state NEW,ESTABLISHED -j ACCEPT
-A OUTPUT -o eth0 -p tcp --sport 22 -m state --state NEW,ESTABLISHED -j ACCEPT

# Open port for https (Splunk website)
-A INPUT -i eth0 -p tcp --dport 443 -m state --state NEW,ESTABLISHED -j ACCEPT
-A OUTPUT -o eth0 -p tcp --sport 443 -m state --state NEW,ESTABLISHED -j ACCEPT

# Open port for splunkd services
-A INPUT -i eth0 -p tcp --dport 8089 -m state --state NEW,ESTABLISHED -j ACCEPT
-A OUTPUT -o eth0 -p tcp --sport 8089 -m state --state NEW,ESTABLISHED -j ACCEPT

# Open port for Splunk Forwarders
-A INPUT -i eth0 -p tcp --dport 9997 -m state --state NEW,ESTABLISHED -j ACCEPT
-A OUTPUT -o eth0 -p tcp --sport 9997 -m state --state NEW,ESTABLISHED -j ACCEPT

COMMIT

And the output of the "iptables -L" command:

Chain INPUT (policy DROP)
target     prot opt source               destination
ACCEPT     icmp --  anywhere             anywhere
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:ssh state NEW,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:https state NEW,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:8089 state NEW,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:palace-6 state NEW,ESTABLISHED

Chain FORWARD (policy DROP)
target     prot opt source               destination

Chain OUTPUT (policy DROP)
target     prot opt source               destination
ACCEPT     tcp  --  anywhere             anywhere            tcp spt:ssh state NEW,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere            tcp spt:https state NEW,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere            tcp spt:8089 state NEW,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere            tcp spt:palace-6 state NEW,ESTABLISHED

I did notice the 9997 port turned into palace-6 under the "iptables -L" command, but I do not think it is causing my issues since I am not running any forwarders yet.

Please let me know if you need any additional information to help with my issues and thank in advance for any help.

I found some additional information related to my problem.

I was looking through the web_access logs and noticed there is a huge difference between the response times when the iptables are enabled and disabled.

With iptables enables, it talkes 60,000+ ms to throw a 503:

10.111.x.xx - admin [24/Nov/2012:20:07:14.070 -0800] "GET /en-US/ HTTP/1.1" 503 1186 "" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20100101 Firefox/16.0" - 50b19972111ca8c50 60085ms

With iptables disabled and 44 ms to go through two redirects and throw a 200 for the login:

10.111.x.xx - admin [24/Nov/2012:20:11:40.128 -0800] "GET / HTTP/1.1" 303 102 "" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20100101 Firefox/16.0" - 50b19a7c207f0d4c10bf90 6ms
10.111.x.xx - admin [24/Nov/2012:20:11:40.140 -0800] "GET /en-US/ HTTP/1.1" 303 124 "" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20100101 Firefox/16.0" - 50b19a7c237f0d4c098310 8ms
10.111.x.xx - admin [24/Nov/2012:20:11:40.152 -0800] "GET /en-US/account/login?return_to=%2Fen-US%2F HTTP/1.1" 200 2848 "" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20100101 Firefox/16.0" - 50b19a7c267f0d4c10b7d0 30ms

[enno]

FYI, the "palace-6" label appearing around the place comes from /etc/services where that's the protocol your OS supplier thinks uses port 9997. If you want you can edit /etc/services and change the name to splunk-f or something similar as I'm yet to see anything that is looking for the palace-6 name on my systems. This will fix any program using the C library getservbyport entry point, so netstat, tcpdump, lsof and so on. N.B. this edit may get overwritten on an OS update in the future.

[jsmander · Accepted solution]

Hi

I've taken a simple approach which you may consider. Create allow entries by host IP and then deny the entire network at the end. As firewall rules stop on matches this has worked. I did not try locking down by port number.

sudo iptables -A INPUT -i eth0 -s 10.10.10.220/32 -j ACCEPT

sudo iptables -A INPUT -i eth0 -s 10.10.10.249/32 -j ACCEPT

sudo iptables -A INPUT -i eth0 -s 10.10.6.90/32 -j ACCEPT

sudo iptables -A INPUT -i eth0 -s 10.0.0.0/8 -j DROP

sudo iptables -A INPUT -i eth0 -s 172.16.0.0/16 -j DROP

sudo iptables -A INPUT -i eth0 -s 192.168.0.0/16 -j DROP

Previously I had tried the 3 accept entries followed by DROP on the INPUT policy but that did not work. So far the above has worked.

Regards
J

[eandresen]

I have not had a chance to work out all of the details, but your basic idea is working for me. Once I get the last few things worked out, I will post a copy of my iptables for others to see. Thanks for the help!

[eandresen]

I will give this "simple approach" idea a try in my setup and update the thread if it works. Unfortunately, my setup is not very simple, so I will have to tweak the idea a bit.

[DaveSavage]

I like that J. Elegant and simple. The server that Splunk is running on may not be dedicated to just that service so this is catered for.

[lguinn2]

By default, the web interface to Splunk runs on port 8000. If you want it to run on port 443 instead, you will need to update the following in web.conf

# Change the default port number:
httpport = 443

# Turn on SSL:
enableSplunkWebSSL = true
# absolute paths may be used here.
privKeyPath = etc/auth/splunkweb/privkey.pem
caCertPath = etc/auth/splunkweb/cert.pem

Note that this is using the Splunk self-generated certificates. For more info, look here.

[eandresen]

Yep, I ran the following commands to enable SSL and update the port to 443 for the Splunk web interface and I am accessing it through https.

• ./splunk enable web-ssl
• ./splunk set web-port 443

So the port differences between what Splunk is using and what is open in the iptables is not the issue.