All Posts

Find Answers
Ask questions. Get answers. Find technical product solutions from passionate members of the Splunk community.

All Posts

Hi @PickleRick, Thank you so much for your help...Please find the comments inline: 1. I assume (never used it myself) that Amazon Linux is also an RPM-based distro and you'll be installing Splunk... See more...
Hi @PickleRick, Thank you so much for your help...Please find the comments inline: 1. I assume (never used it myself) that Amazon Linux is also an RPM-based distro and you'll be installing Splunk the same way it was installed before. Yes, Amazon Linux natively supports RPM package installer 2. Remember to shut down Splunk service before moving the data. And of course don't start the new instance before you copy the data. Got it. 3. I'm not sure why you want to snapshot the volumes. For backup in case you need to roll back? Yes, correct..in case there is a need to rollback 4. You might have other dependencies lying around, not included in $SPLUNK_HOME - for example certificates. In our case, the ssl certificates are deployed under /opt/splunk/etc/certs/ as the ssl offloading is directly on the server and there is no loadbalancer or proxy in the front.  Can you think of anything else that may  deployed outside of /opt/splunk 5. If you move whole filesystems between server instances the UIDs and GIDs might not match and you might need to fix your accesses. Can we recursively chown the files on the new server after migration to ensure correct ownership, hope that should take care of it sudo chown -R splunk:splunk /opt/splunk Oh, and most importantly - I didn't notice that at first - DON'T UPGRADE AND MOVE AT THE SAME TIME! Either upgrade and then do the move to the same version on a new server or move to the same 8.x you have now and then upgrade on the new server. Sure I prefer doing the latter, but the older version of Splunk Enterprise 8.2.2.1 does not support Amazon Linux.
Talked to my sysadmin, we decided to use port 1035 instead of port 514. not getting the socket errors in splunkd.log anymore, but still not seeing the messages from the UF in Splunk Cloud.   root@NH... See more...
Talked to my sysadmin, we decided to use port 1035 instead of port 514. not getting the socket errors in splunkd.log anymore, but still not seeing the messages from the UF in Splunk Cloud.   root@NHC-NETSplunkForwarder:/opt/splunkforwarder/var/log/splunk# cat splunkd.log | grep "1035" 06-26-2025 20:05:00.017 +0000 INFO TcpInputConfig [1851 TcpListener] - IPv4 port 1035 is reserved for raw input 06-26-2025 20:05:00.017 +0000 INFO TcpInputConfig [1851 TcpListener] - IPv4 port 1035 will negotiate s2s protocol level 7 06-26-2025 20:05:00.017 +0000 INFO TcpInputProc [1851 TcpListener] - Creating raw Acceptor for IPv4 port 1035 with Non-SSL 06-26-2025 20:25:30.471 +0000 WARN AutoLoadBalancedConnectionStrategy [1869 TcpOutEloop] - Possible duplication of events with channel=source::udp:1035|host::10.12.2.149|NETWORK|, streamId=1989559377486376685, offset=6 on host=3.213.185.213:9997 connid 0
Yes, that looks like a viable approach. Thank you. Too bad Power Automate is tricky and I'm not a programmer. I'll leave this discussion open for a few in case anyone has already achieved the goal an... See more...
Yes, that looks like a viable approach. Thank you. Too bad Power Automate is tricky and I'm not a programmer. I'll leave this discussion open for a few in case anyone has already achieved the goal and wants to share.
setcap 'cap_net_bind_service=+ep' /opt/splunkforwarder/bin/splunk I just tried this, still seeing the same issue. I also had my system admin move user splunkfwd (this user runs splunk) into the sud... See more...
setcap 'cap_net_bind_service=+ep' /opt/splunkforwarder/bin/splunk I just tried this, still seeing the same issue. I also had my system admin move user splunkfwd (this user runs splunk) into the sudo group  still seeing the same errors in splunkd.log 06-26-2025 18:46:46.515 +0000 INFO TcpInputConfig [921 TcpListener] - IPv4 port 514 is reserved for raw input 06-26-2025 18:46:46.515 +0000 INFO TcpInputConfig [921 TcpListener] - IPv4 port 514 will negotiate s2s protocol level 7 06-26-2025 18:46:46.515 +0000 ERROR TcpInputProc [921 TcpListener] - Could not bind to port IPv4 port 514: Permission denied 06-26-2025 19:27:32.285 +0000 INFO TcpInputConfig [1554 TcpListener] - IPv4 port 514 is reserved for raw input 06-26-2025 19:27:32.286 +0000 INFO TcpInputConfig [1554 TcpListener] - IPv4 port 514 will negotiate s2s protocol level 7 06-26-2025 19:27:32.286 +0000 ERROR TcpInputProc [1554 TcpListener] - Could not bind to port IPv4 port 514: Permission denied  
To allow the UF access to port 514, try this setcap 'cap_net_bind_service=+ep' /path/to/uf
This is... bad, Firstly, it seems that it's data already received by something else, embedded in another format and sent to Splunk. Then secondly, these are completely different sourcetypes. So if ... See more...
This is... bad, Firstly, it seems that it's data already received by something else, embedded in another format and sent to Splunk. Then secondly, these are completely different sourcetypes. So if you absolutely cannot separate them earlier, you should overwrite sourcetype on ingestion so that each of those types is parsed differently.
I've been creating some new modern playbooks in SOAR for automation. One of the playbooks that I created has a drop down next to it that shows an "outputs" menu with Name, Data Type, and Description ... See more...
I've been creating some new modern playbooks in SOAR for automation. One of the playbooks that I created has a drop down next to it that shows an "outputs" menu with Name, Data Type, and Description fields that are all blank. Only one playbook has this option and all were created from scratch. What caused this output dropdown on the one playbook? The playbook type was created as automation and not input.
Please provide more examples of the events you are dealing with, and include your desired results, and what you are getting (and why it is not correct)?
And, to add to already provided answers, there is no such thing as syslog meaning a strictly defined protocol. Syslog can mean many different things depending on context and it's definitely not limit... See more...
And, to add to already provided answers, there is no such thing as syslog meaning a strictly defined protocol. Syslog can mean many different things depending on context and it's definitely not limited to 514 port. It's a perfectly normal situation when "syslog" data is sent to another port.
Heavy forwarder with httpout to indexer cluster - Splunk Community httpout is not a HEC output (although it needs an HEC input and valid HEC token; it's complicated). It's s2s protocol embedded ... See more...
Heavy forwarder with httpout to indexer cluster - Splunk Community httpout is not a HEC output (although it needs an HEC input and valid HEC token; it's complicated). It's s2s protocol embedded in http transport. It is indeed a fairly recent invention mostly aimed at situations like yours - where it's easier (politically, not technically) to allow outgoing http traffic (even if it's only pseudo-http) than some unknown protocol. Maybe, this is the correct explanation.
We will be installing Splunk Connect 4 Syslog soon. But I haven't got there yet. That will be more involved. We previously tried running syslog-ng on the server and monitoring the file, but everythi... See more...
We will be installing Splunk Connect 4 Syslog soon. But I haven't got there yet. That will be more involved. We previously tried running syslog-ng on the server and monitoring the file, but everything came into splunk cloud from the same host in Splunk Cloud. It was a mess. When I installed the Universal Forwarder on the new servers, I created new user splunkfwd to run it, just like the instructions said. Can I simply change the permissions for user splunkfwd? At this point I don't really care if it runs with root privileges. what would the needed permissions for user splunkfwd to overcome this? Thanks, -Pete 
Let me clarify terms and be more specific: S2S+TLS = Splunk to Splunk Protocol with TLS Encryption HTTPS = HTTP Protocol with TLS Encryption I would like to use the HTTP protocol with TLS to send ... See more...
Let me clarify terms and be more specific: S2S+TLS = Splunk to Splunk Protocol with TLS Encryption HTTPS = HTTP Protocol with TLS Encryption I would like to use the HTTP protocol with TLS to send data from a Heavy Forwarder to a HTTP Event Collector (HEC). There are configuration options in the outputs.conf spec for doing this. This post also says something similar: How to send data to two output types, [tcpout] and... - Splunk Community "It also states httpout is only supported on UFs but it works on HFs as well. I've tested with both httpout and tcpout but httpout will take precedence every-time." From everything I can tell, it never works.  It doesn't even make an attempt to connect to the HEC (verified via packet capture).
Those error messages are saying Splunk does not have permission to use port 514.  All ports <1024 are "privileged" and require special permission to access.  Running Splunk as root will solve that, b... See more...
Those error messages are saying Splunk does not have permission to use port 514.  All ports <1024 are "privileged" and require special permission to access.  Running Splunk as root will solve that, but I highly discourage that. The recommended practice is to send syslog data to a dedicated syslog receiver (syslog-ng, for example), have it write the data to disk, and have a UF monitor those disk files.  You also can use Splunk Connect 4 Syslog (SC4S) to send the data directly to Splunk.
Adding on to @livehybrid's response, sending TCP/UDP directly to a Splunk instance is discouraged.  The reason is any time that instance restarts data is lost.  Also, the usual distance between the d... See more...
Adding on to @livehybrid's response, sending TCP/UDP directly to a Splunk instance is discouraged.  The reason is any time that instance restarts data is lost.  Also, the usual distance between the data source and Splunk increases the chances of UDP data getting dropped.
Hi @danielbb  I suspect the main reason for this is that 9514 is not a Privileged port, ie Splunk can mount it (ports > 1024) without additional permissions. To mount a port <1024 Splunk would requi... See more...
Hi @danielbb  I suspect the main reason for this is that 9514 is not a Privileged port, ie Splunk can mount it (ports > 1024) without additional permissions. To mount a port <1024 Splunk would require CAP_NET_BIND_SERVICE capability. It is common practice for Splunk to listen to ports higher than 1024 for syslog, and people often prefix 514 with another number. Sometimes you will see multiples such as 7514,8514,9514 to receive traffic from different syslog sources.  Did this answer help you? If so, please consider: Adding karma to show it was useful Marking it as the solution if it resolved your issue Commenting if you need any clarification Your feedback encourages the volunteers in this community to continue contributing
@livehybrid Again, forgive me if you get repeated replies from me. My replies are not showing after I post them. I'm brand new to the community so maybe I'm missing something silly. To answer your q... See more...
@livehybrid Again, forgive me if you get repeated replies from me. My replies are not showing after I post them. I'm brand new to the community so maybe I'm missing something silly. To answer your questions, sudo netstat -tulnp | grep 514 this returns nothing However, plenty of errors in splunkd.log root@NHC-NETSplunkForwarder:/opt/splunkforwarder/var/log/splunk# cat splunkd.log | grep "514" 06-25-2025 19:24:20.190 +0000 INFO TcpInputConfig [59254 TcpListener] - IPv4 port 514 is reserved for raw input 06-25-2025 19:24:20.190 +0000 INFO TcpInputConfig [59254 TcpListener] - IPv4 port 514 will negotiate s2s protocol level 7 06-25-2025 19:24:20.190 +0000 ERROR TcpInputProc [59254 TcpListener] - Could not bind to port IPv4 port 514: Permission denied 06-25-2025 19:26:21.991 +0000 INFO TcpInputConfig [59507 TcpListener] - IPv4 port 514 is reserved for raw input 06-25-2025 19:26:21.991 +0000 INFO TcpInputConfig [59507 TcpListener] - IPv4 port 514 will negotiate s2s protocol level 7 06-25-2025 19:26:21.992 +0000 ERROR TcpInputProc [59507 TcpListener] - Could not bind to port IPv4 port 514: Permission denied 06-25-2025 21:18:16.827 +0000 INFO TcpInputConfig [60127 TcpListener] - IPv4 port 514 is reserved for raw input 06-25-2025 21:18:16.827 +0000 INFO TcpInputConfig [60127 TcpListener] - IPv4 port 514 will negotiate s2s protocol level 7 06-25-2025 21:18:16.828 +0000 ERROR TcpInputProc [60127 TcpListener] - Could not bind to port IPv4 port 514: Permission denied 06-26-2025 01:38:09.514 +0000 INFO AutoLoadBalancedConnectionStrategy [60145 TcpOutEloop] - Connected to idx=34.201.206.231:9997:0, pset=0, reuse=0. using ACK. autoBatch=1 06-26-2025 14:41:49.984 +0000 INFO TcpInputConfig [63678 TcpListener] - IPv4 port 514 is reserved for raw input 06-26-2025 14:41:49.984 +0000 INFO TcpInputConfig [63678 TcpListener] - IPv4 port 514 will negotiate s2s protocol level 7 06-26-2025 14:41:49.984 +0000 ERROR TcpInputProc [63678 TcpListener] - Could not bind to port IPv4 port 514: Permission denied
Amazing! thank you.
I came across in our repo a monitoring stanza for f5, which is [UDP://9514]. I wonder if there is any reason not to use syslog for this case, are there any limitations using syslog vs. direct UDP con... See more...
I came across in our repo a monitoring stanza for f5, which is [UDP://9514]. I wonder if there is any reason not to use syslog for this case, are there any limitations using syslog vs. direct UDP connection? Why would anybody bypass syslog?
Thank you, this is exactly what I need.
Hi @LOP22456  The user (typically splunkfwd) that is created is a standard system user, so will be stored in /etc/passwd with other local users and I dont think a password is set, so its not possibl... See more...
Hi @LOP22456  The user (typically splunkfwd) that is created is a standard system user, so will be stored in /etc/passwd with other local users and I dont think a password is set, so its not possible to login with the user. The password would be stored in /etc/shadow if set. Check out https://docs.splunk.com/Documentation/Forwarder/latest/Forwarder/Installanixuniversalforwarder#:~:text=of%20Splunk%20Enterprise.-,Install%20the%20universal%20forwarder%20on%20Linux,-About%20the%20splunkfwd for more information around this if you havent already seen.  Did this answer help you? If so, please consider: Adding karma to show it was useful Marking it as the solution if it resolved your issue Commenting if you need any clarification Your feedback encourages the volunteers in this community to continue contributing