Hi,
I've installed splunk app for Infrastructure on my local PC with Windows10 and want to collect local metrics and logs in this app.
When I configure my local pc as entity I get to a point where a can copy/paste a script in powershell.
When I do so a get:
[*] Install Splunk Universal Forwarder on localhost
[*] indexer server: localhost:9997
[*] checking for previous installations of splunk>...
[!] install directory already exists. continuing to congure ..
Test-Connection : Testing connection to computer 'KR9162NBN' failed: Unknown error (0x2b2a)
At C:\WINDOWS\system32\install_uf_script.ps1:174 char:12
+ $ip_info = Test-Connection -ComputerName $env:computername -count 1 | ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : ResourceUnavailable: (KR9162NBN:String) [Test-Connection], PingException
+ FullyQualifiedErrorId : TestConnectionException,Microsoft.PowerShell.Commands.TestConnectionCommand
[*] configuring metrics & log inputs...
[*] Restarting splunk> universal fowarder
SplunkForwarder: Stopped
Splunk> Needle. Haystack. Found.
Checking prerequisites...
Checking mgmt port [8090]: open
Checking conf files for problems...
Done
Checking default conf files for edits...
Validating installed files against hashes from 'C:\Program Files\SplunkUniversalForwarder\splunkforwarder-7.1.2-a0c72a66db66-windows-64-manifest'
All installed files intact.
Done
All preliminary checks passed.
Starting splunk server daemon (splunkd)...
SplunkForwarder: Starting (pid 22324)
Done
[*] splunk> successfully started.
[*] running clean up.
[*] clean up complete. Exiting...
I previusly had installed a forwarder, so the script uses the existing one, and tries to configure it to sens metrics and logs, but I get the above error and no entity is shown in the App.
When I look at Monitoring Console -> Forwarders I see that the forwarder is up and runing.
Tried to uninstal and reinstal the forwarder - same issue.
Any ideas? Thanks in advance.
Hi,
Seems like Test-Connection failed to get ip_info which is added as dimension. It should still work.
Could you please check: 'SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\local\outputs.conf'. Check if your 'server = ...' setting is correct and you can ping the server.
Also, Can you post your inputs.conf file? 'SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\local\inputs.conf'
so the outputs.conf file
[tcpout]
defaultGroup = default-autolb-group
[tcpout:default-autolb-group]
server = localhost:9997
and i get:
ping localhost
Pinging **** [::1] with 32 bytes of data:
General failure.
General failure.
General failure.
General failure.
and the inputs.conf file
# *** Configure Metrics & Logs collected ***
[perfmon://CPU Load]
counters = % C1 Time;% C2 Time;% Idle Time;% Processor Time;% User Time;% Privileged Time;% Reserved Time;% Interrupt Time
instances = *
interval = 30
object = Processor
index = em_metrics
_meta = os::"Microsoft Windows 10 Pro" os_version::10.0.16299 ip::"" entity_type::Windows_Host
[perfmon://Physical Disk]
counters = % Disk Read Time;% Disk Write Time
instances = *
interval = 30
object = PhysicalDisk
index = em_metrics
_meta = os::"Microsoft Windows 10 Pro" os_version::10.0.16299 ip::"" entity_type::Windows_Host
[perfmon://Network Interface]
counters = Bytes Received/sec;Bytes Sent/sec;Packets Received/sec;Packets Sent/sec;Packets Received Errors;Packets Outbound Errors
instances = *
interval = 30
object = Network Interface
index = em_metrics
_meta = os::"Microsoft Windows 10 Pro" os_version::10.0.16299 ip::"" entity_type::Windows_Host
[perfmon://Available Memory]
counters = Cache Bytes;% Committed Bytes In Use;Page Reads/sec;Pages Input/sec;Pages Output/sec;Committed Bytes;Available Bytes
interval = 30
object = Memory
index = em_metrics
_meta = os::"Microsoft Windows 10 Pro" os_version::10.0.16299 ip::"" entity_type::Windows_Host
It seems you might have issue with your Windows system. If I google "ping localhost general failure", I can see many results with solutions to fix it. You might have to try that to fix it. Let me know if it still doesn't work.
Yeah, I'm trying to figure right now what is going on
Are all of the required ports open and accessible on your Splunk Insight instance? No firewall or network restrictions
Yes they are.
I've done some researching and managed to ping 127.0.0.1.
Done testing:
PS C:\WINDOWS\system32> Test-NetConnection -ComputerName 127.0.0.1 -Port 9997
ComputerName : 127.0.0.1
RemoteAddress : 127.0.0.1
RemotePort : 9997
InterfaceAlias : Loopback Pseudo-Interface 1
SourceAddress : 127.0.0.1
TcpTestSucceeded : True
outputs.conf
[tcpout]
defaultGroup = default-autolb-group
[tcpout:default-autolb-group]
server = 127.0.0.1:9997
Still no Entities visible.
Could you try this CLI command and see if you have any active forwards?
Go to C:\Program Files\SplunkUniversalForwarder\bin and do
".\splunk list forward-server"
If you don't have any user account created. You can follow this to create an account:
https://docs.splunk.com/Documentation/Splunk/7.1.2/Installation/StartSplunkforthefirsttime
"Create administrator credentials manually"
Got an active one as localhost:
PS C:\Program Files\SplunkUniversalForwarder\bin> ./splunk list forward-server
Active forwards:
localhost:9997
Configured but inactive forwards:
None
Just a thought, maybe It has something to do with the field hostname.
I searched the index=_internal and splunk found a field host=xxx
The forwarder on the other hand sends data to 127.0.0.1:9997
outputs.conf:
[tcpout]
defaultGroup = default-autolb-group
[tcpout:default-autolb-group]
server = 127.0.0.1:9997
Checked the indexer:
C:\Program Files\Splunk\bin>splunk display listen
Receiving is enabled on port 9997.
Checked theforwarder again:
C:\Program Files\SplunkUniversalForwarder\bin>splunk list forward-server
Active forwards:
localhost:9997
Configured but inactive forwards:
None
also checked on the indexer splunkd.log:
07-31-2018 10:03:34.311 +0200 INFO TcpOutputProc - Connected to idx=127.0.0.1:9997, pset=0, reuse=0.
07-31-2018 10:03:40.892 +0200 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-winevtlog.exe"" splunk-winevtlog - WinEventMon::configure: Failed to find Event Log with channel name='Forwarded Events'
on forwarder:
C:\Program Files\SplunkUniversalForwarder\bin>splunk show default-hostname
Default hostname for data inputs: xxx.
on receiver GUI:
index=_internal sourcetype=splunkd component=TcpInputConfig OR (host=xxx component=StatusMgr)
gives zero events
I think hostname should not be a problem. You can change it using:
https://answers.splunk.com/answers/154999/how-can-i-change-the-default-hostname-in-splunk.html
Can I see your props and transforms.conf file located in etc\apps\splunk_app_infrastructure\default\ ?
ALso, Can you try this search:
| mstats count where host=* AND metric_name=* by index,host,metric_name
Here you go:
transforms.conf
########### Entity Store #################
[em_entities]
external_type = kvstore
collection = em_entities
fields_list = _key,title,state,dimensions,identifier_dimensions,informational_dimensions,imported_date,updated_date, collectors
[em_collector_configs]
external_type = kvstore
collection = em_collector_configs
fields_list = _key,name,title,source_predicate,title_dimension,identifier_dimensions,informational_dimensions,blacklisted_dimensions,monitoring_frequency,monitoring_lag,monitoring_calculation_window,disabled,vital_metrics
[em_groups]
external_type = kvstore
collection = em_groups
fields_list = _key, name, title, filter
[em_thresholds]
external_type = kvstore
collection = em_thresholds
fields_list = _key, name, type_id, type, metric_name, info_min, info_max, warning_min, warning_max, critical_min, critical_max, email_enabled, email_to, email_when
########### Metrics ######################
[metrics-hostoverride]
DEST_KEY = MetaData:Host
REGEX = host=(\S+)
FORMAT = host::$1
########### Transforms for Windows ######################
[value]
REGEX = .*Value=(\S+).*
FORMAT = _value::$1
WRITE_META = true
# Example: object=PhysicalDisk counter="%_Disk_Write_Time"
# Transform - metric_name::PhysicalDisk.%_Disk_Write_Time
[perfmon_metric_name]
REGEX = .*object=(\S+).*counter=(\S+).*
FORMAT = metric_name::$1.$2 metric_type::$1
WRITE_META = true
[instance]
REGEX = .*instance=(\S+).*
FORMAT = instance::$1
WRITE_META = true
as for the "mstats" command, zero results
Checked also Settings -> indexes to find out if there are any metrics indexes, and there is the only one: "em_metrics" assigned to "splunk_app_infrastructure with 0 event count
if you are an existing Splunk customer please file a support case so we can pick up some more details about your environment.
Have you tried these troubleshooting docs ?:
http://docs.splunk.com/Documentation/Splunk/7.1.2/Troubleshooting/AdvancedWindowsTroubleshooting
http://docs.splunk.com/Documentation/Splunk/latest/Troubleshooting/Cantfinddata
Hi, I'll have to have a closer look on those troubleshootig docs.
you can all try to uninstall completely and reinstall Splunk Universal Forwarder (using the powershell script on Add Data page)
You ran the install script as an administrator, right?