Check SELinux: grep "denied" /var/log/audit/audit.log ... View more

It sounds like you want to monitor on CentOS7: `/var/log/secure` as that file has all the entries for SSH sessions. To make things less complicated: run the SUF as `root`, it will be easier to understand how things work. You can test on an isolated EC2 instance and secure later. If you want to forward `/var/log/secure` from other EC2 instances to your indexer, those will only require a SUF installed. You do not need a SUF and indexer installed on the same host. If you have installed Splunk proper, with web and all you can add a monitor input for `/var/log/secure`. ... View more

I know this sounds bad: disable all unneeded services except Splunk and reboot. ... View more

Check if the webserver is listening: ss -tnl | grep 8000 LISTEN 0 128 *:8000 *:* ... View more

You could also set the host here: [monitor:///var/log/messages] disabled = false host = <FQDN> sourcetype = syslog ... View more

LOL! SELinux again. grep "denied" /var/log/audit/audit.log ... View more

As usual always check SELinux and perms on `/var/run/docker.sock`. ... View more

If you have made many changes and want to restore SELinux contexts and can't remember all the dirs, you can: touch /.autorelabel reboot ... View more

According to the log: your Splunk webserver was listening on loopback only. If you're connecting from localhost to localhost on 8000 that's okay. If not: ncat -v saysplunk 8000 Ncat: Version 7.50 ( https://nmap.org/ncat ) Ncat: Connected to <IP>:8000. ... View more

Check your logs on the SUF and the receiver/indexer. Easiest place to start. Is the SUF connecting and forwarding in the log? Are you receiving and indexing in the log? From the SUF to receiver in `var/log/splunk/splunkd.log`: 08-23-2021 15:02:18.711 +0000 INFO AutoLoadBalancedConnectionStrategy [1037 TcpOutEloop] - Found currently active indexer. Connected to idx=10.10.50.206:9997, reuse=1. On the receiver in `var/log/splunk/metrics.log`: 08-23-2021 15:04:16.198 +0000 INFO Metrics - group=tcpin_connections, 10.10.50.209:37444:9997, connectionType=cooked, sourcePort=37444, sourceHost=10.10.50.209, sourceIp=10.10.50.209, destPort=9997, kb=15.651, _tcp_Bps=517.002, _tcp_KBps=0.505, _tcp_avg_thruput=12.605, _tcp_Kprocessed=41409.860, _tcp_eps=0.323, _process_time_ms=0, evt_misc_kBps=0.000, evt_raw_kBps=0.452, evt_fields_kBps=0.032, evt_fn_kBps=0.000, evt_fv_kBps=0.000, evt_fn_str_kBps=0.000, evt_fn_meta_dyn_kBps=0.000, evt_fn_meta_predef_kBps=0.000, evt_fn_meta_str_kBps=0.000, evt_fv_num_kBps=0.000, evt_fv_str_kBps=0.000, evt_fv_predef_kBps=0.000, evt_fv_offlen_kBps=0.000, evt_fv_fp_kBps=0.000, build=ddff1c41e5cf, version=8.2.1, os=Linux, arch=x86_64, hostname=spl-suf-01.ephemeric.lan, guid=752B1D7C-B3C5-43D3-A779-EB6C0FCAA965, fwdType=uf, ssl=false, lastIndexer=10.10.50.206:9997, ack=false  On the SUF: ncat -v spl-idx-01.ephemeric.lan 9997 Ncat: Version 7.50 ( https://nmap.org/ncat ) Ncat: Connected to 10.10.50.206:9997.   Metrics on the SUF: 08-23-2021 15:12:21.012 +0000 INFO Metrics - group=tcpout_connections, name=default-autolb-group:10.10.50.206:9997:0, sourcePort=8089, destIp=10.10.50.206, destPort=9997, _tcp_Bps=526.97, _tcp_KBps=0.51, _tcp_avg_thruput=11.06, _tcp_Kprocessed=41656, _tcp_eps=0.33, kb=15.44   ... View more

Using unreliable UDP is a last resort. I would recommend using a SUF (Splunk Universal Forwarder) installed on your syslog host. Add a monitor input on said host to read whatever log files you require (/var/log/*) and connect to TCP 9997 on the indexer. Add a forward-server to your SUF, that way you use TCP with TLS if required. You can also enable `UseAck` in the SUF `outputs.conf` for audit grade event collection. If UDP is a must: you said you're running Splunk as root so port 514 UDP should work. Check your splunkd.log for errors. Check metrics.log for data. Make sure no other process is using port 514. Try:   lsof -nP -iUDP:514 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME rsyslogd 30071 root 3u IPv4 11494632 0t0 UDP *:514 rsyslogd 30071 root 4u IPv6 11494633 0t0 UDP *:514 ss -tnlp | grep rsyslogd LISTEN 0 25 *:514 *:* users:(("rsyslogd",pid=30071,fd=5)) LISTEN 0 25 [::]:514 [::]:* users:(("rsyslogd",pid=30071,fd=6))     ... View more

grep "denied" /var/log/audit/audit.log ... View more

Check your SELinux log:   grep "denied" /var/log/audit/audit.log     Enable SELinux perms for your bind mount:   docker run -d -p 8000:8000 -e SPLUNK_START_ARGS='--accept-license' -e SPLUNK_PASSWORD='<password>' -v /opt/splunk:/opt/splunk:Z splunk/splunk:latest     https://docs.docker.com/storage/bind-mounts/#configure-the-selinux-label "The Z option indicates that the bind mount content is private and unshared."   ... View more

Check for any SELinux denies: grep "denied" /var/log/audit/audit.log ... View more

On CentOS 7.9: $> systemctl list-unit-files | grep -i splunk splunkforwarder.service enabled  Package: splunkforwarder-8.2.1-ddff1c41e5cf-linux-2.6-x86_64.rpm   ... View more

One should not need to to: sudo setfacl -m u:splunk:rwx /opt  Not required at all. Please never do `chmod 0777 /dir` either. This is not a DAC write perms issue in `/opt`. This is a SELinux issue. When the homedir creation "fails", files from `/etc/skel` are not copied into `/opt/splunk` so your `.bashrc` et al. are missing. ... View more

Have to agree with @dwaddle, use `semanage` and `restorecon` as `chcon` is not persistent across reboots. On CentOS7/8, there should be no need to change SELinux policy if Splunk is in `/opt/splunk` and binding to standard ports. Only the homedir in `/opt/splunk` will fail to be created during a rpm install as per error message but will still be created in the end. See "non-default homedir location" online for aforementioned error. ... View more

As root: `semanage port -l | grep 8443` and check the output. `grep "8443" /var/log/audit/audit.log`. If you get "denied" on port 8443 in the log, there is the problem. Splunk is not allowed to bind to port 8443 as per policy. One can fix that easily enough. ... View more

Always check the Security Policy before disabling SELinux! The above link to the SELinux policy, IMHO, is not the best. They run Splunk as root. Simply not running Splunk as root and disabling SELinux is far better IMHO. The current Splunk rpm does not fully support SELinux as there is an issue with the non-standard homedir location in `/opt/splunk`. The easiest thing to do is put SELinux in permissive mode and check what is denied. Work from there. It's not hard. Dan Walsh is so helpful and so is the community. Don't muck about with commands, just do: `grep "denied" /var/log/audit/audit.log`. ... View more

I had the same problem. The problem is a homedir in a non-standard location: https://www.unixmen.com/selinux-and-non-default-home-directory-locations/ $> ls -dlZ /home/robertg drwx------. robertg robertg unconfined_u:object_r:user_home_dir_t:s0 /home/robertg $> ls -dlZ /opt/splunk drwxr-xr-x. splunk splunk unconfined_u:object_r:usr_t:s0 /opt/splunk The above link explains in detail but the actual fix is: #> semanage fcontext -a -e /home/robertg /opt/splunk #This updates the labels of /opt/splunk, with the –R option doing it recursively for existing subdirectories. #> restorecon –R /opt/splunk #> ls -ldZ /opt/splunk drwxr-xr-x. splunk splunk unconfined_u:object_r:user_home_dir_t:s0 /opt/splunk   If installing from RPM, see what creates the homedir: $> rpm -q --scripts splunk   You can temporarily disable SELinux (not recommended): #> setenforce 0 commands... #> setenforce 1   Check any denied AVCs in `/var/log/audit/audit.log`. Generally speaking: an app in its own dir like in /opt/appname is okay with SELinux as long as it writes there alone. Listening on ports below 1024 etc. requires additional SELinux work. The current security in deploying to /opt/splunk using an interactive login account is wrong IMHO. The account should be a system account (UID <1000) with no login like a classic UNIX daemon account. See `/etc/passwd` for examples. I have tested the above and it works. Use Systemd to manage the service as `splunk:splunk` user,  group respectively. ... View more

I would say a passwd file is different in the way that it provides more options like mail address etc. as opposed to the seed file. But in the end same but different. ... View more

"The "splunktcp" input is not a "data" input, but instead an input to listen to Splunk Forwarders." Thank you, this clarified the problem for me as I was experiencing the same issue. ... View more

Hi, I needed to know the same answer so I setup Splunk 6.4.8 and tested. To my dismay all buckets from all indexes got to your coldToFrozenDir as db_*. Tried: coldToFrozenDir = /media/archive/splunk/$_index_name but it created /media/archive/splunk/\$_index_name/ Lame. ... View more