All Posts

Find Answers
Ask questions. Get answers. Find technical product solutions from passionate members of the Splunk community.

All Posts

Those error messages are saying Splunk does not have permission to use port 514.  All ports <1024 are "privileged" and require special permission to access.  Running Splunk as root will solve that, b... See more...
Those error messages are saying Splunk does not have permission to use port 514.  All ports <1024 are "privileged" and require special permission to access.  Running Splunk as root will solve that, but I highly discourage that. The recommended practice is to send syslog data to a dedicated syslog receiver (syslog-ng, for example), have it write the data to disk, and have a UF monitor those disk files.  You also can use Splunk Connect 4 Syslog (SC4S) to send the data directly to Splunk.
Adding on to @livehybrid's response, sending TCP/UDP directly to a Splunk instance is discouraged.  The reason is any time that instance restarts data is lost.  Also, the usual distance between the d... See more...
Adding on to @livehybrid's response, sending TCP/UDP directly to a Splunk instance is discouraged.  The reason is any time that instance restarts data is lost.  Also, the usual distance between the data source and Splunk increases the chances of UDP data getting dropped.
Hi @danielbb  I suspect the main reason for this is that 9514 is not a Privileged port, ie Splunk can mount it (ports > 1024) without additional permissions. To mount a port <1024 Splunk would requi... See more...
Hi @danielbb  I suspect the main reason for this is that 9514 is not a Privileged port, ie Splunk can mount it (ports > 1024) without additional permissions. To mount a port <1024 Splunk would require CAP_NET_BIND_SERVICE capability. It is common practice for Splunk to listen to ports higher than 1024 for syslog, and people often prefix 514 with another number. Sometimes you will see multiples such as 7514,8514,9514 to receive traffic from different syslog sources.  Did this answer help you? If so, please consider: Adding karma to show it was useful Marking it as the solution if it resolved your issue Commenting if you need any clarification Your feedback encourages the volunteers in this community to continue contributing
@livehybrid Again, forgive me if you get repeated replies from me. My replies are not showing after I post them. I'm brand new to the community so maybe I'm missing something silly. To answer your q... See more...
@livehybrid Again, forgive me if you get repeated replies from me. My replies are not showing after I post them. I'm brand new to the community so maybe I'm missing something silly. To answer your questions, sudo netstat -tulnp | grep 514 this returns nothing However, plenty of errors in splunkd.log root@NHC-NETSplunkForwarder:/opt/splunkforwarder/var/log/splunk# cat splunkd.log | grep "514" 06-25-2025 19:24:20.190 +0000 INFO TcpInputConfig [59254 TcpListener] - IPv4 port 514 is reserved for raw input 06-25-2025 19:24:20.190 +0000 INFO TcpInputConfig [59254 TcpListener] - IPv4 port 514 will negotiate s2s protocol level 7 06-25-2025 19:24:20.190 +0000 ERROR TcpInputProc [59254 TcpListener] - Could not bind to port IPv4 port 514: Permission denied 06-25-2025 19:26:21.991 +0000 INFO TcpInputConfig [59507 TcpListener] - IPv4 port 514 is reserved for raw input 06-25-2025 19:26:21.991 +0000 INFO TcpInputConfig [59507 TcpListener] - IPv4 port 514 will negotiate s2s protocol level 7 06-25-2025 19:26:21.992 +0000 ERROR TcpInputProc [59507 TcpListener] - Could not bind to port IPv4 port 514: Permission denied 06-25-2025 21:18:16.827 +0000 INFO TcpInputConfig [60127 TcpListener] - IPv4 port 514 is reserved for raw input 06-25-2025 21:18:16.827 +0000 INFO TcpInputConfig [60127 TcpListener] - IPv4 port 514 will negotiate s2s protocol level 7 06-25-2025 21:18:16.828 +0000 ERROR TcpInputProc [60127 TcpListener] - Could not bind to port IPv4 port 514: Permission denied 06-26-2025 01:38:09.514 +0000 INFO AutoLoadBalancedConnectionStrategy [60145 TcpOutEloop] - Connected to idx=34.201.206.231:9997:0, pset=0, reuse=0. using ACK. autoBatch=1 06-26-2025 14:41:49.984 +0000 INFO TcpInputConfig [63678 TcpListener] - IPv4 port 514 is reserved for raw input 06-26-2025 14:41:49.984 +0000 INFO TcpInputConfig [63678 TcpListener] - IPv4 port 514 will negotiate s2s protocol level 7 06-26-2025 14:41:49.984 +0000 ERROR TcpInputProc [63678 TcpListener] - Could not bind to port IPv4 port 514: Permission denied
Amazing! thank you.
I came across in our repo a monitoring stanza for f5, which is [UDP://9514]. I wonder if there is any reason not to use syslog for this case, are there any limitations using syslog vs. direct UDP con... See more...
I came across in our repo a monitoring stanza for f5, which is [UDP://9514]. I wonder if there is any reason not to use syslog for this case, are there any limitations using syslog vs. direct UDP connection? Why would anybody bypass syslog?
Thank you, this is exactly what I need.
Hi @LOP22456  The user (typically splunkfwd) that is created is a standard system user, so will be stored in /etc/passwd with other local users and I dont think a password is set, so its not possibl... See more...
Hi @LOP22456  The user (typically splunkfwd) that is created is a standard system user, so will be stored in /etc/passwd with other local users and I dont think a password is set, so its not possible to login with the user. The password would be stored in /etc/shadow if set. Check out https://docs.splunk.com/Documentation/Forwarder/latest/Forwarder/Installanixuniversalforwarder#:~:text=of%20Splunk%20Enterprise.-,Install%20the%20universal%20forwarder%20on%20Linux,-About%20the%20splunkfwd for more information around this if you havent already seen.  Did this answer help you? If so, please consider: Adding karma to show it was useful Marking it as the solution if it resolved your issue Commenting if you need any clarification Your feedback encourages the volunteers in this community to continue contributing  
UF it is. Not HF.
Thanks! Which user should I be logged in as to run 'splunk offline', root or splunk?
Hi @braxton839  This looks good to me - Check the cluster manager first to ensure that things are in a good shape before offline-ing your indexer. Be aware that with a RF of 2 you can only safely t... See more...
Hi @braxton839  This looks good to me - Check the cluster manager first to ensure that things are in a good shape before offline-ing your indexer. Be aware that with a RF of 2 you can only safely take down a single indexer at a time, and whilst it is down you are at risk of reduced data availability if your other indexers run into any problems. Theres some good reading at https://help.splunk.com/en/splunk-enterprise/administer/manage-indexers-and-indexer-clusters/9.3/manage-the-indexer-cluster/take-a-peer-offline if it helps, but you may have already seen this.  Did this answer help you? If so, please consider: Adding karma to show it was useful Marking it as the solution if it resolved your issue Commenting if you need any clarification Your feedback encourages the volunteers in this community to continue contributing
Hello, I have a request from a systems manager related to SOX controls. They are requesting information around the local Splunk account that is created when a UF is being installed (this is on a Lin... See more...
Hello, I have a request from a systems manager related to SOX controls. They are requesting information around the local Splunk account that is created when a UF is being installed (this is on a Linux machine). They are asking where the password is stored for this account/who has access to it, and what are the controls around it. They are requesting to make this account non-interactive - would this cause any problems? They would then have to go around to all 200+ UFs and do this, not sure how intuitive this would be. Has anyone encountered requests related to local Splunk UF accounts & SOX controls?
Hi @danielbb  You could start with something like this: index=_audit action=search info=completed search_et!="N/A" | eval time_span=search_lt-search_et | eval time_span_group=case( time_span<3... See more...
Hi @danielbb  You could start with something like this: index=_audit action=search info=completed search_et!="N/A" | eval time_span=search_lt-search_et | eval time_span_group=case( time_span<3600, "<1hr", time_span>=3600 AND time_span<7200, "1-2hrs", time_span>=7200 AND time_span<43200, "<12hrs", time_span>=43200 AND time_span<86400, "<24hrs", time_span>=86400 AND time_span<259200, "<3days", time_span>=259200 AND time_span<604800, "<7days", time_span>=604800 AND time_span<2592000, "<30days", time_span>=2592000 AND time_span<7776000, "<90days", time_span>=7776000 AND time_span<31536000, "<1year", time_span>=31536000, "more" ) | stats count by time_span_group  Did this answer help you? If so, please consider: Adding karma to show it was useful Marking it as the solution if it resolved your issue Commenting if you need any clarification Your feedback encourages the volunteers in this community to continue contributing
Check out the search_et and search_lt fields.
We would like to produce statistics about the usage of Splunk and we would like to categorize the searches by ranges, whether they cover the last day, past week or past month, and I wonder which fiel... See more...
We would like to produce statistics about the usage of Splunk and we would like to categorize the searches by ranges, whether they cover the last day, past week or past month, and I wonder which fields in _audit provide the beginning and end interval of the search.  
Hello @livehybrid  looks like the json is from Vector agent to Kafka, that's why we may end up with json or is it possible to convert json to raw log in Splunk?
Hi @Pete_  The splunkclouduf.spl app configures secure forwarding to Splunk Cloud; you should not need to modify outputs.conf directly, also, because you're able to see the new forwarders in the Clo... See more...
Hi @Pete_  The splunkclouduf.spl app configures secure forwarding to Splunk Cloud; you should not need to modify outputs.conf directly, also, because you're able to see the new forwarders in the Cloud Monitoring Console (CMC) we know that the outputs are established and the new UFs can reach Splunk Cloud. The testing you've done shows the 514 syslog feed arriving at the box, however is Splunk listening on that port? If you run the following can you see that splunkd is listening to the port? sudo netstat -tulnp | grep 514 Are there any logs in $SPLUNK_HOME/var/log/splunk/splunkd.log about binding port 514, any errors etc?  Did this answer help you? If so, please consider: Adding karma to show it was useful Marking it as the solution if it resolved your issue Commenting if you need any clarification Your feedback encourages the volunteers in this community to continue contributing
I think I know how to do this but I thought it would be best to check with some of the experts here first.   I am upgrading the hardware (storage expansion) on our indexers and this will require tu... See more...
I think I know how to do this but I thought it would be best to check with some of the experts here first.   I am upgrading the hardware (storage expansion) on our indexers and this will require turning off and unplugging each device. Indexers are clustered with a Replication Factor of 2. From what I have read: I can issue the 'splunk offline' command on the indexer I am working on Wait for the indexer to wrap up any tasks Then shut down and unplug the machine to perform this upgrade Once complete, I can plug it back in and turn it back on (make sure Splunk starts running again) Am i missing anything important? Thanks!
Hello, I am having issues getting data into Splunk Cloud with two new Universal forwarders. I have two existing Universal Forwarders that are working just fine, but I am migrating these to new serv... See more...
Hello, I am having issues getting data into Splunk Cloud with two new Universal forwarders. I have two existing Universal Forwarders that are working just fine, but I am migrating these to new servers. Same Universal Forwarder version on both the old and new servers (9.4.3) I have the Universal Forwader software installed on both the new Linux servers. I copied the inputs.conf and outputs.conf files from the old servers. I also installed splunkclouduf.spl that I downloaded from my Splunk Cloud instance. The usage for these forwarders is limited to syslog messages only. I receive syslog messages from other devices on port 514 of the Universal Forwarders (UDP and TCP allowed) and those messages forward to Splunk Cloud. Pretty simple setup. I have confirmed that traffic is being received on the servers on port 514 using tcpdump. However, none of that traffic is reaching Splunk Cloud. I can see the new forwarders in the Splunk Cloud Monitoring Console under Forwarders->Versions and Forwarders->Instance. But no data is being received from the new forwarders. Below are my inputs.conf and outputs.conf files from one of the new servers. As you can see, very simple setup and outputs.conf is doing nothing. Again, these were copied from my old working servers exactly, except for the hostname on the new forwarders. ---------------------------------------- inputs.conf  [default] host = NHC-NETSplunkForwarder [tcp://514] acceptFrom = * connection_host=ip index=nhcnetwork sourcetype=NETWORK disabled=0 [udp://514] acceptFrom = * connection_host=ip index=nhcnetwork sourcetype=NETWORK ---------------------------------------- outputs.conf (sanitized) #This breaks stuff. The credentials package provides what is needed here. Leave commented out. #[tcpout] #defaultGroup = splunkcloud,default-autolb-group #[tcpout:default-autolb-group] #server = XXXXXXX.splunkcloud.com:9997 #disabled = false #[tcpout-server://XXXXXXX.splunkcloud.com:9997] Do I need to do something in Splunk Cloud to allow these new forwarders to send data? I don't know how splunkclouduf.spl works so I don't know a way to monitor output traffic from the Universal Forwarder. Any suggestions or tips are appreciated. Thanks, -Pete  
Can you also confirm, is the data coming from a UF? I saw you put that the conf was on the Indexers but if its being sent from a Heavy Forwarder it will need to be there too. Is this a regular monit... See more...
Can you also confirm, is the data coming from a UF? I saw you put that the conf was on the Indexers but if its being sent from a Heavy Forwarder it will need to be there too. Is this a regular monitor:// input?  Did this answer help you? If so, please consider: Adding karma to show it was useful Marking it as the solution if it resolved your issue Commenting if you need any clarification Your feedback encourages the volunteers in this community to continue contributing