I don't. I have not checked back on this topic in quite some time. ... View more

I think the OP wants to secure the UFs. By default the UF binds `*:8089` which is an audit finding in most envs. To be sure, configure /opt/splunkforwarder/etc/splunk-launch.conf: SPLUNK_BINDIP=127.0.0.1   ... View more

For RHEL7/CentOS7: /etc/default/grub.conf GRUB_CMDLINE_LINUX="... transparent_hugepage=never" Make your GRUB config... Or non-persistent whilst running: yum install libhugetlbfs-utils.x86_64 hugeadm --thp-never   ... View more

You could change in `etc/splunk-launch.conf`:   SPLUNK_DB=/tmp/splunk_db    Just to see if you can get a clean start? I'm probably wrong but it looks like there is a problem binding to port 8000? This could be caused by other errors, like a filesystem problem? I haven't looked at errors like this for years as I am only now making a comeback to the world of Splunk. ... View more

08-26-2021 16:21:40.575 -0800 INFO AutoLoadBalancedConnectionStrategy [12416 TcpOutEloop] - Found currently active indexer. Connected to idx=192.168.0.21:9997, reuse=1. It has connected. ... View more

$> telnet 192.168.0.21 9997 ... View more

It is not recommended to run a UF and indexer/search head on a single VM but for testing and learning it might work. As the log shows: your UF has connected to the IDX. That is correct: your UF is not configured as a deployment client as you are not using a deployment server. Nothing wrong here. An example check for UF and IDX: # UF. $> splunk list forward-server Active forwards: spl-idx-01.ephemeric.lan:9997 # IDX. $> ss -ant | grep 9997 ESTAB 170697 0 10.10.50.206:9997 10.10.50.209:44208   ... View more

Yes, copy the directory to the forwarders etc/apps. ... View more

Just copy the app to etc/apps on your Splunk image. Same thing as deploying said app. Just be forewarned: you lose the ability to do any updates after deployment using a DS. If there are any required changes... problem. You can also use a config management tool like Chef, Ansible, Puppet etc. If networking is an issue: have the indexer(s) and DS on the same host as that then only requires a single dest IP to ports 8089, 9997. ... View more

May I suggest using https://github.com/OpenVPN/easy-rsa? The concept is like so: - you generate a keypair for the client (SUF) - you generate a keypair for the server (receiver, indexer etc.) - the client and server each have the same CA cert So three files on the server and three files on the client in a single file. This is the simplest setup. ... View more

Sounds like https://en.wikipedia.org/wiki/XY_problem ... View more

May we know what file this is? I would recommend against unpacking and repacking the rpm as pointed out. One can build a rpm but the process is complicated and *.spec files are convoluted at best. If I know the file you wish to include, which should be deployed via some other mechanism like Ansible, Chef, Puppet, I can offer advice. You can do a script with the rpm inline as an option. ... View more

I can only find: https://docs.splunk.com/Documentation/VersionCompatibility/current/Matrix/Compatibilitybetweenforwardersandindexers WRT compatibility. Can anyone confirm compatibility between UFs and DSs? ... View more

Seems like your connection is broken somehow. You said that "some" of the UFs are failing. I think you meant apps are pulled from the DS, not pushed from the DS to UFs. Can you check your firewall log for any errors? Please try from your failing UF: echo '\n' | openssl s_client -connect <deployment-server>:8089   ... View more

Please search for "discard events" on Answers. Many posts explaining how to do what you require. ... View more

I like it! Well said. ... View more

Why do you need to run Splunk in a container? It complicates everything. First, test your ulimit settings in a VM to see how and why things work the way they do. Remove as much as you can. Simplify. If you must use Docker, use the `--ulimit` option: $> grep -i limit /usr/lib/systemd/system/docker.service LimitNOFILE=1048576 LimitNPROC=1048576 LimitCORE=infinity $> ulimit -n 1024 $> docker run -it --ulimit nofile=64000:64000 alpine / # ulimit -n 64000 / #   ... View more

It also depends on your network topography. The TA must be able to connect to said REST API. There is "search time" and "index time" and if your TA does the latter, it belongs on an indexer. Nothing precludes your TA being installed on a SUF and writing to a file waiting to be forwarded... ... View more

Do not hardcode IP addresses in prod. In dev, sure, for testing. If you change your IPs once in prod it becomes unwieldy. Use DNS. In your case: you haven't stated the hostname you're using so we assume it is colliding with the public DNS namespace. Change your hostnames to a private namespace or if you must use FQDNs in the public namespace, put entries in `/etc/hosts`: ## # Host Database # # localhost is used to configure the loopback interface # when the system is booting. Do not change this entry. ## 127.0.0.1 localhost 255.255.255.255 broadcasthost ::1 localhost 192.168.20.149 idx01.host.tld 192.168.20.148 idx02.host.tld ... View more

IMHO: I think there is some confusion here. The OP wants to ingest logs on the host via the SUF. So permissions for `splunk` need to be granted on, for example `/var/log/messages` either via group or `setfacl`. ... View more

We tried this setup a long time ago: had our DS deploy an app with a deploymentclient.conf so we could control the DS location. All was good until we decided to migrate our DS to another host. The new DS was accidentally started without any apps in place. The behaviour of Splunk at the time (unsure if it's any different now): all SUF clients connected, saw there were no apps, proceeded to delete all of their apps on the localhost! So your deploymentclient.conf is now gone and we lost 400 SUFs as they were trying to connect with no config. We had an amazing team and client and we recovered 80% of the SUFs in 24 hours! I would suggest using DNS to control IP changes to your DS etc. PS You can edit etc/system/local/* as we did by deploying an app with a script that makes edits to those files (sed and awk). Same danger here though: one mistake and it's over. Do Canary Releases! ... View more

Check your logs. Check your logs. Check your logs. ... View more