topic Splunk Running with Firewall/iptables Enabled in Deployment Architecture

Splunk Running with Firewall/iptables Enabled

eandresen — Sat, 24 Nov 2012 08:09:32 GMT

I am experiencing some strange issues with Splunk running properly and be responsive while the Linux Firewall/IPTables are enabled.

When the iptables service is running, Splunk is unresponsive to CLI commands through SSH and the webpage throws a "503 - Service Unavailable" following an attempted login with the admin account. I soon as I stop the iptables services, everything works fine. I can issue commands like "./splunk show splunkd-port" and "./splunk show web-port" without problems and logging into the webpage works great.

Below are the basics of my setup:

Running CentOS 6.3 (fully updated) on a VMWare server.
Running Splunk 5.0.1 version.

Below are the Splunk ports I am using:

Web port: 443 (SSL enabled)
Splunkd port: 8089
Splunk Forwarder: 9997

Below are the settings within the "/etc/sysconfig/iptables" file:

# Firewall configuration written by system-config-firewall
# Manual customization of this file is not recommended.
*filter :INPUT DROP [0:0] :OUTPUT DROP [0:0] :FORWARD DROP [0:0]

# Open port for ping
-A INPUT -p icmp -j ACCEPT

# Open port for SSH
-A INPUT -i eth0 -p tcp --dport 22 -m state --state NEW,ESTABLISHED -j ACCEPT
-A OUTPUT -o eth0 -p tcp --sport 22 -m state --state NEW,ESTABLISHED -j ACCEPT

# Open port for https (Splunk website)
-A INPUT -i eth0 -p tcp --dport 443 -m state --state NEW,ESTABLISHED -j ACCEPT
-A OUTPUT -o eth0 -p tcp --sport 443 -m state --state NEW,ESTABLISHED -j ACCEPT

# Open port for splunkd services
-A INPUT -i eth0 -p tcp --dport 8089 -m state --state NEW,ESTABLISHED -j ACCEPT
-A OUTPUT -o eth0 -p tcp --sport 8089 -m state --state NEW,ESTABLISHED -j ACCEPT

# Open port for Splunk Forwarders
-A INPUT -i eth0 -p tcp --dport 9997 -m state --state NEW,ESTABLISHED -j ACCEPT
-A OUTPUT -o eth0 -p tcp --sport 9997 -m state --state NEW,ESTABLISHED -j ACCEPT

COMMIT

And the output of the "iptables -L" command:

Chain INPUT (policy DROP)
target     prot opt source               destination
ACCEPT     icmp --  anywhere             anywhere
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:ssh state NEW,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:https state NEW,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:8089 state NEW,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:palace-6 state NEW,ESTABLISHED

Chain FORWARD (policy DROP)
target     prot opt source               destination

Chain OUTPUT (policy DROP)
target     prot opt source               destination
ACCEPT     tcp  --  anywhere             anywhere            tcp spt:ssh state NEW,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere            tcp spt:https state NEW,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere            tcp spt:8089 state NEW,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere            tcp spt:palace-6 state NEW,ESTABLISHED

I did notice the 9997 port turned into palace-6 under the "iptables -L" command, but I do not think it is causing my issues since I am not running any forwarders yet.

Please let me know if you need any additional information to help with my issues and thank in advance for any help.

I found some additional information related to my problem.

I was looking through the web_access logs and noticed there is a huge difference between the response times when the iptables are enabled and disabled.

With iptables enables, it talkes 60,000+ ms to throw a 503:

10.111.x.xx - admin [24/Nov/2012:20:07:14.070 -0800] "GET /en-US/ HTTP/1.1" 503 1186 "" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20100101 Firefox/16.0" - 50b19972111ca8c50 60085ms

With iptables disabled and 44 ms to go through two redirects and throw a 200 for the login:

10.111.x.xx - admin [24/Nov/2012:20:11:40.128 -0800] "GET / HTTP/1.1" 303 102 "" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20100101 Firefox/16.0" - 50b19a7c207f0d4c10bf90 6ms
10.111.x.xx - admin [24/Nov/2012:20:11:40.140 -0800] "GET /en-US/ HTTP/1.1" 303 124 "" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20100101 Firefox/16.0" - 50b19a7c237f0d4c098310 8ms
10.111.x.xx - admin [24/Nov/2012:20:11:40.152 -0800] "GET /en-US/account/login?return_to=%2Fen-US%2F HTTP/1.1" 200 2848 "" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20100101 Firefox/16.0" - 50b19a7c267f0d4c10b7d0 30ms

Re: Splunk Running with Firewall/iptables Enabled

lguinn2 — Sun, 25 Nov 2012 01:56:58 GMT

By default, the web interface to Splunk runs on port 8000. If you want it to run on port 443 instead, you will need to update the following in web.conf

# Change the default port number:
httpport = 443

# Turn on SSL:
enableSplunkWebSSL = true
# absolute paths may be used here.
privKeyPath = etc/auth/splunkweb/privkey.pem
caCertPath = etc/auth/splunkweb/cert.pem

Note that this is using the Splunk self-generated certificates. For more info, look here.

Re: Splunk Running with Firewall/iptables Enabled

eandresen — Sun, 25 Nov 2012 04:27:07 GMT

Yep, I ran the following commands to enable SSL and update the port to 443 for the Splunk web interface and I am accessing it through https.

./splunk enable web-ssl
./splunk set web-port 443

So the port differences between what Splunk is using and what is open in the iptables is not the issue.

Re: Splunk Running with Firewall/iptables Enabled

jsmander — Tue, 27 Nov 2012 14:26:40 GMT

I've taken a simple approach which you may consider. Create allow entries by host IP and then deny the entire network at the end. As firewall rules stop on matches this has worked. I did not try locking down by port number.

sudo iptables -A INPUT -i eth0 -s 10.10.10.220/32 -j ACCEPT

sudo iptables -A INPUT -i eth0 -s 10.10.10.249/32 -j ACCEPT

sudo iptables -A INPUT -i eth0 -s 10.10.6.90/32 -j ACCEPT

sudo iptables -A INPUT -i eth0 -s 10.0.0.0/8 -j DROP

sudo iptables -A INPUT -i eth0 -s 172.16.0.0/16 -j DROP

sudo iptables -A INPUT -i eth0 -s 192.168.0.0/16 -j DROP

Previously I had tried the 3 accept entries followed by DROP on the INPUT policy but that did not work. So far the above has worked.

Regards
J

Re: Splunk Running with Firewall/iptables Enabled

DaveSavage — Tue, 27 Nov 2012 16:50:43 GMT

I like that J. Elegant and simple. The server that Splunk is running on may not be dedicated to just that service so this is catered for.

Re: Splunk Running with Firewall/iptables Enabled

eandresen — Wed, 28 Nov 2012 17:15:40 GMT

I will give this "simple approach" idea a try in my setup and update the thread if it works. Unfortunately, my setup is not very simple, so I will have to tweak the idea a bit.

Re: Splunk Running with Firewall/iptables Enabled

eandresen — Sat, 08 Dec 2012 03:08:24 GMT

I have not had a chance to work out all of the details, but your basic idea is working for me. Once I get the last few things worked out, I will post a copy of my iptables for others to see. Thanks for the help!

Re: Splunk Running with Firewall/iptables Enabled

enno — Tue, 25 Aug 2015 02:31:42 GMT

FYI, the "palace-6" label appearing around the place comes from /etc/services where that's the protocol your OS supplier thinks uses port 9997. If you want you can edit /etc/services and change the name to splunk-f or something similar as I'm yet to see anything that is looking for the palace-6 name on my systems. This will fix any program using the C library getservbyport entry point, so netstat, tcpdump, lsof and so on. N.B. this edit may get overwritten on an OS update in the future.