Splunk App for Infrastructure - forwarder issue

[danielwysockiar]

Hi,
I've installed splunk app for Infrastructure on my local PC with Windows10 and want to collect local metrics and logs in this app.
When I configure my local pc as entity I get to a point where a can copy/paste a script in powershell.

When I do so a get:

[*] Install Splunk Universal Forwarder on localhost
[*] indexer server: localhost:9997
[*] checking for previous installations of splunk>...
[!] install directory already exists. continuing to congure ..
Test-Connection : Testing connection to computer 'KR9162NBN' failed: Unknown error (0x2b2a)
At C:\WINDOWS\system32\install_uf_script.ps1:174 char:12
+ $ip_info = Test-Connection -ComputerName $env:computername -count 1 | ...
+            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ResourceUnavailable: (KR9162NBN:String) [Test-Connection], PingException
    + FullyQualifiedErrorId : TestConnectionException,Microsoft.PowerShell.Commands.TestConnectionCommand

[*] configuring metrics & log inputs...
[*] Restarting splunk> universal fowarder
SplunkForwarder: Stopped

Splunk> Needle. Haystack. Found.

Checking prerequisites...
        Checking mgmt port [8090]: open
        Checking conf files for problems...
        Done
        Checking default conf files for edits...
        Validating installed files against hashes from 'C:\Program Files\SplunkUniversalForwarder\splunkforwarder-7.1.2-a0c72a66db66-windows-64-manifest'
        All installed files intact.
        Done
All preliminary checks passed.

Starting splunk server daemon (splunkd)...

SplunkForwarder: Starting (pid 22324)
Done

[*] splunk> successfully started.
[*] running clean up.
[*] clean up complete. Exiting...

I previusly had installed a forwarder, so the script uses the existing one, and tries to configure it to sens metrics and logs, but I get the above error and no entity is shown in the App.

When I look at Monitoring Console -> Forwarders I see that the forwarder is up and runing.

Tried to uninstal and reinstal the forwarder - same issue.

Any ideas? Thanks in advance.

[dagarwal_splunk]

Hi,
Seems like Test-Connection failed to get ip_info which is added as dimension. It should still work.

Could you please check: 'SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\local\outputs.conf'. Check if your 'server = ...' setting is correct and you can ping the server.

Also, Can you post your inputs.conf file? 'SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\local\inputs.conf'

[danielwysockiar]

so the outputs.conf file

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = localhost:9997

and i get:

ping localhost

Pinging **** [::1] with 32 bytes of data:
General failure.
General failure.
General failure.
General failure.

and the inputs.conf file

# *** Configure Metrics & Logs collected ***
[perfmon://CPU Load]
counters = % C1 Time;% C2 Time;% Idle Time;% Processor Time;% User Time;% Privileged Time;% Reserved Time;% Interrupt Time
instances = *
interval = 30
object = Processor
index = em_metrics
_meta =  os::"Microsoft Windows 10 Pro" os_version::10.0.16299 ip::"" entity_type::Windows_Host


[perfmon://Physical Disk]
counters = % Disk Read Time;% Disk Write Time
instances = *
interval = 30
object = PhysicalDisk
index = em_metrics
_meta =  os::"Microsoft Windows 10 Pro" os_version::10.0.16299 ip::"" entity_type::Windows_Host


[perfmon://Network Interface]
counters = Bytes Received/sec;Bytes Sent/sec;Packets Received/sec;Packets Sent/sec;Packets Received Errors;Packets Outbound Errors
instances = *
interval = 30
object = Network Interface
index = em_metrics
_meta =  os::"Microsoft Windows 10 Pro" os_version::10.0.16299 ip::"" entity_type::Windows_Host


[perfmon://Available Memory]
counters = Cache Bytes;% Committed Bytes In Use;Page Reads/sec;Pages Input/sec;Pages Output/sec;Committed Bytes;Available Bytes
interval = 30
object = Memory
index = em_metrics
_meta =  os::"Microsoft Windows 10 Pro" os_version::10.0.16299 ip::"" entity_type::Windows_Host

[dagarwal_splunk]

It seems you might have issue with your Windows system. If I google "ping localhost general failure", I can see many results with solutions to fix it. You might have to try that to fix it. Let me know if it still doesn't work.

[danielwysockiar]

Yeah, I'm trying to figure right now what is going on

[ntankersley_spl]

Are all of the required ports open and accessible on your Splunk Insight instance? No firewall or network restrictions

[danielwysockiar]

Yes they are.
I've done some researching and managed to ping 127.0.0.1.
Done testing:

PS C:\WINDOWS\system32> Test-NetConnection -ComputerName 127.0.0.1 -Port 9997


ComputerName     : 127.0.0.1
RemoteAddress    : 127.0.0.1
RemotePort       : 9997
InterfaceAlias   : Loopback Pseudo-Interface 1
SourceAddress    : 127.0.0.1
TcpTestSucceeded : True

outputs.conf

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = 127.0.0.1:9997

Still no Entities visible.

[dagarwal_splunk]

Could you try this CLI command and see if you have any active forwards?
Go to C:\Program Files\SplunkUniversalForwarder\bin and do
".\splunk list forward-server"

If you don't have any user account created. You can follow this to create an account:
https://docs.splunk.com/Documentation/Splunk/7.1.2/Installation/StartSplunkforthefirsttime
"Create administrator credentials manually"

[danielwysockiar]

Got an active one as localhost:

    PS C:\Program Files\SplunkUniversalForwarder\bin> ./splunk list forward-server
    Active forwards:
            localhost:9997
    Configured but inactive forwards:
            None

[danielwysockiar]

Just a thought, maybe It has something to do with the field hostname.

I searched the index=_internal and splunk found a field host=xxx

The forwarder on the other hand sends data to 127.0.0.1:9997
outputs.conf:

 [tcpout]
 defaultGroup = default-autolb-group

 [tcpout:default-autolb-group]
 server = 127.0.0.1:9997

Checked the indexer:

C:\Program Files\Splunk\bin>splunk display listen
Receiving is enabled on port 9997.

Checked theforwarder again:

C:\Program Files\SplunkUniversalForwarder\bin>splunk list forward-server
Active forwards:
        localhost:9997
Configured but inactive forwards:
        None

also checked on the indexer splunkd.log:

07-31-2018 10:03:34.311 +0200 INFO  TcpOutputProc - Connected to idx=127.0.0.1:9997, pset=0, reuse=0.
07-31-2018 10:03:40.892 +0200 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-winevtlog.exe"" splunk-winevtlog - WinEventMon::configure: Failed to find Event Log with channel name='Forwarded Events'

on forwarder:

C:\Program Files\SplunkUniversalForwarder\bin>splunk show default-hostname
    Default hostname for data inputs: xxx.

on receiver GUI:

index=_internal sourcetype=splunkd component=TcpInputConfig OR (host=xxx component=StatusMgr)

gives zero events

[dagarwal_splunk]

I think hostname should not be a problem. You can change it using:
https://answers.splunk.com/answers/154999/how-can-i-change-the-default-hostname-in-splunk.html

Can I see your props and transforms.conf file located in etc\apps\splunk_app_infrastructure\default\ ?

ALso, Can you try this search:

| mstats count where host=* AND metric_name=* by index,host,metric_name

[danielwysockiar]

Here you go:
transforms.conf

########### Entity Store #################
[em_entities]
external_type = kvstore
collection = em_entities
fields_list = _key,title,state,dimensions,identifier_dimensions,informational_dimensions,imported_date,updated_date, collectors

[em_collector_configs]
external_type = kvstore
collection = em_collector_configs
fields_list = _key,name,title,source_predicate,title_dimension,identifier_dimensions,informational_dimensions,blacklisted_dimensions,monitoring_frequency,monitoring_lag,monitoring_calculation_window,disabled,vital_metrics

[em_groups]
external_type = kvstore
collection = em_groups
fields_list = _key, name, title, filter

[em_thresholds]
external_type = kvstore
collection = em_thresholds
fields_list = _key, name, type_id, type, metric_name, info_min, info_max, warning_min, warning_max, critical_min, critical_max, email_enabled, email_to, email_when

########### Metrics ######################
[metrics-hostoverride]
DEST_KEY = MetaData:Host
REGEX = host=(\S+)
FORMAT = host::$1

########### Transforms for Windows ######################
[value]
REGEX = .*Value=(\S+).*
FORMAT = _value::$1
WRITE_META = true

# Example: object=PhysicalDisk counter="%_Disk_Write_Time"
# Transform - metric_name::PhysicalDisk.%_Disk_Write_Time
[perfmon_metric_name]
REGEX = .*object=(\S+).*counter=(\S+).*
FORMAT = metric_name::$1.$2 metric_type::$1
WRITE_META = true

[instance]
REGEX = .*instance=(\S+).*
FORMAT = instance::$1
WRITE_META = true

as for the "mstats" command, zero results

Checked also Settings -> indexes to find out if there are any metrics indexes, and there is the only one: "em_metrics" assigned to "splunk_app_infrastructure with 0 event count

[dagarwal_splunk]

if you are an existing Splunk customer please file a support case so we can pick up some more details about your environment.

Have you tried these troubleshooting docs ?:
http://docs.splunk.com/Documentation/Splunk/7.1.2/Troubleshooting/AdvancedWindowsTroubleshooting
http://docs.splunk.com/Documentation/Splunk/latest/Troubleshooting/Cantfinddata

[danielwysockiar]

Hi, I'll have to have a closer look on those troubleshootig docs.

[dagarwal_splunk]

you can all try to uninstall completely and reinstall Splunk Universal Forwarder (using the powershell script on Add Data page)

[dagarwal_splunk]

You ran the install script as an administrator, right?