Hi,
We have a Splunk environment with universal forwarders already installed on our Windows servers. We want to try the Splunk App for infrastructure. Can we use the existing Universal Forwarders to use the Splunk App for Infrastructure without having to running the script and reinstalling on all of our servers.
Thanks.
You need to setup inputs.conf (add all metrics and logs data to collect) and outputs.conf (send data to SAI instance) on existing Splunk Universal Forwarders.
Here is useful link:
http://docs.splunk.com/Documentation/InfraApp/1.2.2/Admin/ManualInstallWindowsUF
Something similar to this :
https://answers.splunk.com/answers/699711/can-you-help-me-use-the-splunk-app-for-infrastruct.html#an...
Here is the curl command result:
{"text":"Server is busy","code":9,"invalid-event-number":0}
I didn't see any indexing error like " metric event not indexed". But do see following Parsing error:
01-07-2019 17:18:04.592 -0500 ERROR HttpInputDataHandler - Parsing error : Server is busy
Thanks, this fixed the issue!
Thanks again!
Great! you can add your dimensions back now if you want..
It seems you put inside Plugin write_splunk -
Dimension "entity_type:Linux_Host"
Formatting got messed up when you pasted collectd.conf file here.
Check collectd again:
1. collectd service is running.
Yes
2. No errors in collectd.log file
No
3. Plugin write_splunk is there in collectd.conf file.
server "ServerIpAddress"
port "8088"
token "tokenValve_Not_the name of the token"
ssl true
verifyssl false
here is Customization for Splunk:
server "100.111.111.111"
port "8088"
token "xxxxxxxxxxxxxxxxxxxxxxxxxx"
ssl true
verifyssl false
What about the log file for collectd ? Also, What OS version you are using?
Linux version: 2.6.32-642.15.1.el6.x86_64
Try this search and see if it returns anything:
| mstats avg(_value) WHERE index=em_metrics AND metric_name=cpu.* by host
Also, what is the sourcetype and index for your hec_token? Check hec_token in your Splunk instance that is used by collectd.
Make sure both are "em_metrics".
https://docs.splunk.com/Documentation/InfraApp/1.2.2/Install/Install
See step 5. You can use the Splunk Web. What token are you using right now in collectd.conf file?
I missed SourceType, set to em_metrics now. should I restart the service?
only one token was configured with splunk server.
It should work now. You can try the search again?
might need to restart
I disabled old token, created a new token, restart the server. Then updated the token to the newly created one with agent collectd.conf, then restarted collectd ( no error) and uf, but still nothing returned when I did the search.
Let's try to debug:
Check collectd again:
1. collectd service is running.
2. No errors in collectd.log file
3. Plugin write_splunk is there in collectd.conf file.
4. The server (SAI server), token, port are correct for write_splunk in collectd.conf file
Check SAI again:
1. Add-on for Infra installed
2. Token used by collectd is enabled and use em_metrics for sourcetype and index.
Hostname "xxxxxxx"
FQDNLookup false
Interval 60
WriteQueueLimitHigh 1000000
WriteQueueLimitLow 800000
LoadPlugin syslog
LoadPlugin logfile
FlushInterval 30
LoadPlugin cpu
LoadPlugin memory
LoadPlugin df
LoadPlugin load
LoadPlugin disk
LoadPlugin interface
LogLevel info
File "/etc/collectd/collectd.log"
Timestamp true
PrintSeverity true
LogLevel info
ReportByCpu true
ReportByState true
ValuesPercentage true
ValuesAbsolute false
ValuesPercentage true
FSType "ext2"
FSType "ext3"
FSType "ext4"
FSType "XFS"
FSType "rootfs"
FSType "overlay"
FSType "hfs"
ReportByDevice true
ValuesAbsolute false
ValuesPercentage true
IgnoreSelected false
ReportRelative true
Disk ""
IgnoreSelected true
UdevNameAttr "DEVNAME"
IgnoreSelected true
server "1111111111111111111"
port "8088"
token "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
ssl true
verifyssl false