I would like to monitor 10 hosts on a Splunk server. is that possible? What are the steps to monitor clients or hosts on Splunk server? should i install Universal forwader on all clients? I am confused and looking for suggestions.
Hi ammul440,
at first follow the basic Splunk training and read Splunk documentation to understand how Splunk works.
Anyway, it isn't so immediate:
These are some starting links:
https://docs.splunk.com/Documentation/SplunkCloud/7.2.7/Data/Getstartedwithgettingdatain
https://www.tutorialspoint.com/splunk/index.htm
https://www.youtube.com/watch?v=6lX4DOd1T-s
https://www.youtube.com/watch?v=DJ6tXTsjX_A
https://www.youtube.com/watch?v=ZlKPqjuM0wo
http://www.splunk.com/view/SP-CAAAH9U
https://www.splunk.com/en_us/training/free-courses/splunk-fundamentals-1.html
https://docs.splunk.com/Documentation/Splunk/7.3.2/SearchTutorial/WelcometotheSearchTutorial
Ciao.
Giuseppe
Hi Giuseppe,
Yes, I have read the basic documentation and some of the videos. I installed splunk universal forwarder on one of the client and set the forwarding data using the command, also configured the inputs and outputs.conf
./splunk add monitor /var/log/syslog -sourcetype systemlog
unfortunately, the data is not routing to the splunk server. on the splunk web interface the messages as fllows.
"The TCP output processor has paused the data flow. Forwarding to output group default-autolb-group has been blocked for 10 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data. Learn more."
"Now skipping indexing of internal audit events, because the downstream queue is not accepting data. Will keep dropping events until data flow resumes. Review system health: ensure downstream indexing and/or forwarding are operating correctly. "
please suggest.
Hi ammul440,
At first, di you enabled receiving on Splunkserver?
in [Settings -- Forwarding and Receiving -- Receive Data] choose the port you used on UF to send data (default 9997), probably you have to restart Splunk.
If this doesn't resolve, please share the following files on $SPLUNK_HOME/etc/system/local:
Then some questions:
Bye.
Giuseppe
Hi Giuseppe,
Thanks for your quick response. Yes, I have enabled 9997.
inputs.conf
[default]
[monitor:///var/log/syslog]
sourcetype = systemlog
outputs.conf
[tcpout]
defaultGroup = default-autolb-group
[tcpout:default-autolb-group]
disabled = false
server = 192.168.1.12:9997
[tcpout-server://192.168.1.12:9997]
1) I tested the connection, connection works but i guess port 9997 is closed on the target server and also on recieving server.
2) As a root, i installed splunk UF
Thank you.
Hi ammul440,
probably the problem is on the connection.
If you want to ingest syslogs, use the sourcetype=syslog, so you'll have some default settings on Splunk Server.
Bye.
Giuseppe
I used the iptables to add the port.
opt/splunkforwarder/bin/splunk list forward-server
Your session is invalid. Please login.
Splunk username: admin
Password:
Active forwards:
None
Configured but inactive forwards:
192.168.1.12:9997
where 192.168.1.12 is splunk server . I am not sure whats wrong
Hi ammul440,
did you tested ports using telnet from the target to the Splunk server?
telnet 192.168.1.12 9997
If it's ok, run this search on the Splunk server
index=_internal host=<client_hostname>
if you have logs, it means that the connection is ok but there's an input problem, if you haven't there's still a connection problem.
Then, sorry for the stupid notations:
Ciao.
Giuseppe
Hi please tell me
ok seems there is a telnet issue
Trying ..
telnet: Unable to connect to remote host: Connection refused
when i ran this index=_internal host= i dindot get any output just the command worked..
I have enabled receiving on splunk server
what do you mean by target server (splunk server)?
Aslo, I have a doubt. Do we need to configure Indexer as a separate node?
Hi ammul440,
the error you report says that you have to open firewall routes between targets and Splunk server on port 9997 for data and on 8089 for management (if you use Splunk server also as a Deployment server.
This is always the first check to do before to install a Universal Forwarder.
About the search I suggested, it has the objective to see if the target is connected to the Indexer.
I don't understand when you speak of a "separate node", target and Splunk server are on the same server?
Ciao.
Giuseppe