at first follow the basic Splunk training and read Splunk documentation to understand how Splunk works.
Anyway, it isn't so immediate:
These are some starting links:
Yes, I have read the basic documentation and some of the videos. I installed splunk universal forwarder on one of the client and set the forwarding data using the command, also configured the inputs and outputs.conf
./splunk add monitor /var/log/syslog -sourcetype systemlog
unfortunately, the data is not routing to the splunk server. on the splunk web interface the messages as fllows.
"The TCP output processor has paused the data flow. Forwarding to output group default-autolb-group has been blocked for 10 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data. Learn more."
"Now skipping indexing of internal audit events, because the downstream queue is not accepting data. Will keep dropping events until data flow resumes. Review system health: ensure downstream indexing and/or forwarding are operating correctly. "
At first, di you enabled receiving on Splunkserver?
in [Settings -- Forwarding and Receiving -- Receive Data] choose the port you used on UF to send data (default 9997), probably you have to restart Splunk.
If this doesn't resolve, please share the following files on $SPLUNK_HOME/etc/system/local:
Then some questions:
Thanks for your quick response. Yes, I have enabled 9997.
sourcetype = systemlog
defaultGroup = default-autolb-group
disabled = false
server = 192.168.1.12:9997
1) I tested the connection, connection works but i guess port 9997 is closed on the target server and also on recieving server.
2) As a root, i installed splunk UF
probably the problem is on the connection.
If you want to ingest syslogs, use the sourcetype=syslog, so you'll have some default settings on Splunk Server.
I used the iptables to add the port.
opt/splunkforwarder/bin/splunk list forward-server
Your session is invalid. Please login.
Splunk username: admin
Configured but inactive forwards:
where 192.168.1.12 is splunk server . I am not sure whats wrong
did you tested ports using telnet from the target to the Splunk server?
telnet 192.168.1.12 9997
If it's ok, run this search on the Splunk server
if you have logs, it means that the connection is ok but there's an input problem, if you haven't there's still a connection problem.
Then, sorry for the stupid notations:
ok seems there is a telnet issue
telnet: Unable to connect to remote host: Connection refused
when i ran this index=_internal host= i dindot get any output just the command worked..
I have enabled receiving on splunk server
what do you mean by target server (splunk server)?
Aslo, I have a doubt. Do we need to configure Indexer as a separate node?
the error you report says that you have to open firewall routes between targets and Splunk server on port 9997 for data and on 8089 for management (if you use Splunk server also as a Deployment server.
This is always the first check to do before to install a Universal Forwarder.
About the search I suggested, it has the objective to see if the target is connected to the Indexer.
I don't understand when you speak of a "separate node", target and Splunk server are on the same server?