All Posts

Find Answers
Ask questions. Get answers. Find technical product solutions from passionate members of the Splunk community.

All Posts

Heavy forwarder with httpout to indexer cluster - Splunk Community httpout is not a HEC output (although it needs an HEC input and valid HEC token; it's complicated). It's s2s protocol embedded ... See more...
Heavy forwarder with httpout to indexer cluster - Splunk Community httpout is not a HEC output (although it needs an HEC input and valid HEC token; it's complicated). It's s2s protocol embedded in http transport. It is indeed a fairly recent invention mostly aimed at situations like yours - where it's easier (politically, not technically) to allow outgoing http traffic (even if it's only pseudo-http) than some unknown protocol. Maybe, this is the correct explanation.
We will be installing Splunk Connect 4 Syslog soon. But I haven't got there yet. That will be more involved. We previously tried running syslog-ng on the server and monitoring the file, but everythi... See more...
We will be installing Splunk Connect 4 Syslog soon. But I haven't got there yet. That will be more involved. We previously tried running syslog-ng on the server and monitoring the file, but everything came into splunk cloud from the same host in Splunk Cloud. It was a mess. When I installed the Universal Forwarder on the new servers, I created new user splunkfwd to run it, just like the instructions said. Can I simply change the permissions for user splunkfwd? At this point I don't really care if it runs with root privileges. what would the needed permissions for user splunkfwd to overcome this? Thanks, -Pete 
Let me clarify terms and be more specific: S2S+TLS = Splunk to Splunk Protocol with TLS Encryption HTTPS = HTTP Protocol with TLS Encryption I would like to use the HTTP protocol with TLS to send ... See more...
Let me clarify terms and be more specific: S2S+TLS = Splunk to Splunk Protocol with TLS Encryption HTTPS = HTTP Protocol with TLS Encryption I would like to use the HTTP protocol with TLS to send data from a Heavy Forwarder to a HTTP Event Collector (HEC). There are configuration options in the outputs.conf spec for doing this. This post also says something similar: How to send data to two output types, [tcpout] and... - Splunk Community "It also states httpout is only supported on UFs but it works on HFs as well. I've tested with both httpout and tcpout but httpout will take precedence every-time." From everything I can tell, it never works.  It doesn't even make an attempt to connect to the HEC (verified via packet capture).
Those error messages are saying Splunk does not have permission to use port 514.  All ports <1024 are "privileged" and require special permission to access.  Running Splunk as root will solve that, b... See more...
Those error messages are saying Splunk does not have permission to use port 514.  All ports <1024 are "privileged" and require special permission to access.  Running Splunk as root will solve that, but I highly discourage that. The recommended practice is to send syslog data to a dedicated syslog receiver (syslog-ng, for example), have it write the data to disk, and have a UF monitor those disk files.  You also can use Splunk Connect 4 Syslog (SC4S) to send the data directly to Splunk.
Adding on to @livehybrid's response, sending TCP/UDP directly to a Splunk instance is discouraged.  The reason is any time that instance restarts data is lost.  Also, the usual distance between the d... See more...
Adding on to @livehybrid's response, sending TCP/UDP directly to a Splunk instance is discouraged.  The reason is any time that instance restarts data is lost.  Also, the usual distance between the data source and Splunk increases the chances of UDP data getting dropped.
Hi @danielbb  I suspect the main reason for this is that 9514 is not a Privileged port, ie Splunk can mount it (ports > 1024) without additional permissions. To mount a port <1024 Splunk would requi... See more...
Hi @danielbb  I suspect the main reason for this is that 9514 is not a Privileged port, ie Splunk can mount it (ports > 1024) without additional permissions. To mount a port <1024 Splunk would require CAP_NET_BIND_SERVICE capability. It is common practice for Splunk to listen to ports higher than 1024 for syslog, and people often prefix 514 with another number. Sometimes you will see multiples such as 7514,8514,9514 to receive traffic from different syslog sources.  Did this answer help you? If so, please consider: Adding karma to show it was useful Marking it as the solution if it resolved your issue Commenting if you need any clarification Your feedback encourages the volunteers in this community to continue contributing
@livehybrid Again, forgive me if you get repeated replies from me. My replies are not showing after I post them. I'm brand new to the community so maybe I'm missing something silly. To answer your q... See more...
@livehybrid Again, forgive me if you get repeated replies from me. My replies are not showing after I post them. I'm brand new to the community so maybe I'm missing something silly. To answer your questions, sudo netstat -tulnp | grep 514 this returns nothing However, plenty of errors in splunkd.log root@NHC-NETSplunkForwarder:/opt/splunkforwarder/var/log/splunk# cat splunkd.log | grep "514" 06-25-2025 19:24:20.190 +0000 INFO TcpInputConfig [59254 TcpListener] - IPv4 port 514 is reserved for raw input 06-25-2025 19:24:20.190 +0000 INFO TcpInputConfig [59254 TcpListener] - IPv4 port 514 will negotiate s2s protocol level 7 06-25-2025 19:24:20.190 +0000 ERROR TcpInputProc [59254 TcpListener] - Could not bind to port IPv4 port 514: Permission denied 06-25-2025 19:26:21.991 +0000 INFO TcpInputConfig [59507 TcpListener] - IPv4 port 514 is reserved for raw input 06-25-2025 19:26:21.991 +0000 INFO TcpInputConfig [59507 TcpListener] - IPv4 port 514 will negotiate s2s protocol level 7 06-25-2025 19:26:21.992 +0000 ERROR TcpInputProc [59507 TcpListener] - Could not bind to port IPv4 port 514: Permission denied 06-25-2025 21:18:16.827 +0000 INFO TcpInputConfig [60127 TcpListener] - IPv4 port 514 is reserved for raw input 06-25-2025 21:18:16.827 +0000 INFO TcpInputConfig [60127 TcpListener] - IPv4 port 514 will negotiate s2s protocol level 7 06-25-2025 21:18:16.828 +0000 ERROR TcpInputProc [60127 TcpListener] - Could not bind to port IPv4 port 514: Permission denied 06-26-2025 01:38:09.514 +0000 INFO AutoLoadBalancedConnectionStrategy [60145 TcpOutEloop] - Connected to idx=34.201.206.231:9997:0, pset=0, reuse=0. using ACK. autoBatch=1 06-26-2025 14:41:49.984 +0000 INFO TcpInputConfig [63678 TcpListener] - IPv4 port 514 is reserved for raw input 06-26-2025 14:41:49.984 +0000 INFO TcpInputConfig [63678 TcpListener] - IPv4 port 514 will negotiate s2s protocol level 7 06-26-2025 14:41:49.984 +0000 ERROR TcpInputProc [63678 TcpListener] - Could not bind to port IPv4 port 514: Permission denied
Amazing! thank you.
I came across in our repo a monitoring stanza for f5, which is [UDP://9514]. I wonder if there is any reason not to use syslog for this case, are there any limitations using syslog vs. direct UDP con... See more...
I came across in our repo a monitoring stanza for f5, which is [UDP://9514]. I wonder if there is any reason not to use syslog for this case, are there any limitations using syslog vs. direct UDP connection? Why would anybody bypass syslog?
Thank you, this is exactly what I need.
Hi @LOP22456  The user (typically splunkfwd) that is created is a standard system user, so will be stored in /etc/passwd with other local users and I dont think a password is set, so its not possibl... See more...
Hi @LOP22456  The user (typically splunkfwd) that is created is a standard system user, so will be stored in /etc/passwd with other local users and I dont think a password is set, so its not possible to login with the user. The password would be stored in /etc/shadow if set. Check out https://docs.splunk.com/Documentation/Forwarder/latest/Forwarder/Installanixuniversalforwarder#:~:text=of%20Splunk%20Enterprise.-,Install%20the%20universal%20forwarder%20on%20Linux,-About%20the%20splunkfwd for more information around this if you havent already seen.  Did this answer help you? If so, please consider: Adding karma to show it was useful Marking it as the solution if it resolved your issue Commenting if you need any clarification Your feedback encourages the volunteers in this community to continue contributing  
UF it is. Not HF.
Thanks! Which user should I be logged in as to run 'splunk offline', root or splunk?
Hi @braxton839  This looks good to me - Check the cluster manager first to ensure that things are in a good shape before offline-ing your indexer. Be aware that with a RF of 2 you can only safely t... See more...
Hi @braxton839  This looks good to me - Check the cluster manager first to ensure that things are in a good shape before offline-ing your indexer. Be aware that with a RF of 2 you can only safely take down a single indexer at a time, and whilst it is down you are at risk of reduced data availability if your other indexers run into any problems. Theres some good reading at https://help.splunk.com/en/splunk-enterprise/administer/manage-indexers-and-indexer-clusters/9.3/manage-the-indexer-cluster/take-a-peer-offline if it helps, but you may have already seen this.  Did this answer help you? If so, please consider: Adding karma to show it was useful Marking it as the solution if it resolved your issue Commenting if you need any clarification Your feedback encourages the volunteers in this community to continue contributing
Hello, I have a request from a systems manager related to SOX controls. They are requesting information around the local Splunk account that is created when a UF is being installed (this is on a Lin... See more...
Hello, I have a request from a systems manager related to SOX controls. They are requesting information around the local Splunk account that is created when a UF is being installed (this is on a Linux machine). They are asking where the password is stored for this account/who has access to it, and what are the controls around it. They are requesting to make this account non-interactive - would this cause any problems? They would then have to go around to all 200+ UFs and do this, not sure how intuitive this would be. Has anyone encountered requests related to local Splunk UF accounts & SOX controls?
Hi @danielbb  You could start with something like this: index=_audit action=search info=completed search_et!="N/A" | eval time_span=search_lt-search_et | eval time_span_group=case( time_span<3... See more...
Hi @danielbb  You could start with something like this: index=_audit action=search info=completed search_et!="N/A" | eval time_span=search_lt-search_et | eval time_span_group=case( time_span<3600, "<1hr", time_span>=3600 AND time_span<7200, "1-2hrs", time_span>=7200 AND time_span<43200, "<12hrs", time_span>=43200 AND time_span<86400, "<24hrs", time_span>=86400 AND time_span<259200, "<3days", time_span>=259200 AND time_span<604800, "<7days", time_span>=604800 AND time_span<2592000, "<30days", time_span>=2592000 AND time_span<7776000, "<90days", time_span>=7776000 AND time_span<31536000, "<1year", time_span>=31536000, "more" ) | stats count by time_span_group  Did this answer help you? If so, please consider: Adding karma to show it was useful Marking it as the solution if it resolved your issue Commenting if you need any clarification Your feedback encourages the volunteers in this community to continue contributing
Check out the search_et and search_lt fields.
We would like to produce statistics about the usage of Splunk and we would like to categorize the searches by ranges, whether they cover the last day, past week or past month, and I wonder which fiel... See more...
We would like to produce statistics about the usage of Splunk and we would like to categorize the searches by ranges, whether they cover the last day, past week or past month, and I wonder which fields in _audit provide the beginning and end interval of the search.  
Hello @livehybrid  looks like the json is from Vector agent to Kafka, that's why we may end up with json or is it possible to convert json to raw log in Splunk?
Hi @Pete_  The splunkclouduf.spl app configures secure forwarding to Splunk Cloud; you should not need to modify outputs.conf directly, also, because you're able to see the new forwarders in the Clo... See more...
Hi @Pete_  The splunkclouduf.spl app configures secure forwarding to Splunk Cloud; you should not need to modify outputs.conf directly, also, because you're able to see the new forwarders in the Cloud Monitoring Console (CMC) we know that the outputs are established and the new UFs can reach Splunk Cloud. The testing you've done shows the 514 syslog feed arriving at the box, however is Splunk listening on that port? If you run the following can you see that splunkd is listening to the port? sudo netstat -tulnp | grep 514 Are there any logs in $SPLUNK_HOME/var/log/splunk/splunkd.log about binding port 514, any errors etc?  Did this answer help you? If so, please consider: Adding karma to show it was useful Marking it as the solution if it resolved your issue Commenting if you need any clarification Your feedback encourages the volunteers in this community to continue contributing