All Apps and Add-ons

Splunk App for Infrastructure - forwarder issue

danielwysockiar
Explorer

Hi,
I've installed splunk app for Infrastructure on my local PC with Windows10 and want to collect local metrics and logs in this app.
When I configure my local pc as entity I get to a point where a can copy/paste a script in powershell.

When I do so a get:

[*] Install Splunk Universal Forwarder on localhost
[*] indexer server: localhost:9997
[*] checking for previous installations of splunk>...
[!] install directory already exists. continuing to congure ..
Test-Connection : Testing connection to computer 'KR9162NBN' failed: Unknown error (0x2b2a)
At C:\WINDOWS\system32\install_uf_script.ps1:174 char:12
+ $ip_info = Test-Connection -ComputerName $env:computername -count 1 | ...
+            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ResourceUnavailable: (KR9162NBN:String) [Test-Connection], PingException
    + FullyQualifiedErrorId : TestConnectionException,Microsoft.PowerShell.Commands.TestConnectionCommand

[*] configuring metrics & log inputs...
[*] Restarting splunk> universal fowarder
SplunkForwarder: Stopped

Splunk> Needle. Haystack. Found.

Checking prerequisites...
        Checking mgmt port [8090]: open
        Checking conf files for problems...
        Done
        Checking default conf files for edits...
        Validating installed files against hashes from 'C:\Program Files\SplunkUniversalForwarder\splunkforwarder-7.1.2-a0c72a66db66-windows-64-manifest'
        All installed files intact.
        Done
All preliminary checks passed.

Starting splunk server daemon (splunkd)...

SplunkForwarder: Starting (pid 22324)
Done

[*] splunk> successfully started.
[*] running clean up.
[*] clean up complete. Exiting...

I previusly had installed a forwarder, so the script uses the existing one, and tries to configure it to sens metrics and logs, but I get the above error and no entity is shown in the App.

When I look at Monitoring Console -> Forwarders I see that the forwarder is up and runing.

Tried to uninstal and reinstal the forwarder - same issue.

Any ideas? Thanks in advance.

0 Karma

dagarwal_splunk
Splunk Employee
Splunk Employee

Hi,
Seems like Test-Connection failed to get ip_info which is added as dimension. It should still work.

Could you please check: 'SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\local\outputs.conf'. Check if your 'server = ...' setting is correct and you can ping the server.

Also, Can you post your inputs.conf file? 'SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\local\inputs.conf'

0 Karma

danielwysockiar
Explorer

so the outputs.conf file

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = localhost:9997

and i get:

ping localhost

Pinging **** [::1] with 32 bytes of data:
General failure.
General failure.
General failure.
General failure.

and the inputs.conf file

# *** Configure Metrics & Logs collected ***
[perfmon://CPU Load]
counters = % C1 Time;% C2 Time;% Idle Time;% Processor Time;% User Time;% Privileged Time;% Reserved Time;% Interrupt Time
instances = *
interval = 30
object = Processor
index = em_metrics
_meta =  os::"Microsoft Windows 10 Pro" os_version::10.0.16299 ip::"" entity_type::Windows_Host


[perfmon://Physical Disk]
counters = % Disk Read Time;% Disk Write Time
instances = *
interval = 30
object = PhysicalDisk
index = em_metrics
_meta =  os::"Microsoft Windows 10 Pro" os_version::10.0.16299 ip::"" entity_type::Windows_Host


[perfmon://Network Interface]
counters = Bytes Received/sec;Bytes Sent/sec;Packets Received/sec;Packets Sent/sec;Packets Received Errors;Packets Outbound Errors
instances = *
interval = 30
object = Network Interface
index = em_metrics
_meta =  os::"Microsoft Windows 10 Pro" os_version::10.0.16299 ip::"" entity_type::Windows_Host


[perfmon://Available Memory]
counters = Cache Bytes;% Committed Bytes In Use;Page Reads/sec;Pages Input/sec;Pages Output/sec;Committed Bytes;Available Bytes
interval = 30
object = Memory
index = em_metrics
_meta =  os::"Microsoft Windows 10 Pro" os_version::10.0.16299 ip::"" entity_type::Windows_Host
0 Karma

dagarwal_splunk
Splunk Employee
Splunk Employee

It seems you might have issue with your Windows system. If I google "ping localhost general failure", I can see many results with solutions to fix it. You might have to try that to fix it. Let me know if it still doesn't work.

0 Karma

danielwysockiar
Explorer

Yeah, I'm trying to figure right now what is going on

0 Karma

ntankersley_spl
Splunk Employee
Splunk Employee

Are all of the required ports open and accessible on your Splunk Insight instance? No firewall or network restrictions

0 Karma

danielwysockiar
Explorer

Yes they are.
I've done some researching and managed to ping 127.0.0.1.
Done testing:

PS C:\WINDOWS\system32> Test-NetConnection -ComputerName 127.0.0.1 -Port 9997


ComputerName     : 127.0.0.1
RemoteAddress    : 127.0.0.1
RemotePort       : 9997
InterfaceAlias   : Loopback Pseudo-Interface 1
SourceAddress    : 127.0.0.1
TcpTestSucceeded : True

outputs.conf

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = 127.0.0.1:9997

Still no Entities visible.

0 Karma

dagarwal_splunk
Splunk Employee
Splunk Employee

Could you try this CLI command and see if you have any active forwards?
Go to C:\Program Files\SplunkUniversalForwarder\bin and do
".\splunk list forward-server"

If you don't have any user account created. You can follow this to create an account:
https://docs.splunk.com/Documentation/Splunk/7.1.2/Installation/StartSplunkforthefirsttime
"Create administrator credentials manually"

0 Karma

danielwysockiar
Explorer

Got an active one as localhost:

    PS C:\Program Files\SplunkUniversalForwarder\bin> ./splunk list forward-server
    Active forwards:
            localhost:9997
    Configured but inactive forwards:
            None
0 Karma

danielwysockiar
Explorer

Just a thought, maybe It has something to do with the field hostname.

I searched the index=_internal and splunk found a field host=xxx

The forwarder on the other hand sends data to 127.0.0.1:9997
outputs.conf:

 [tcpout]
 defaultGroup = default-autolb-group

 [tcpout:default-autolb-group]
 server = 127.0.0.1:9997

Checked the indexer:

C:\Program Files\Splunk\bin>splunk display listen
Receiving is enabled on port 9997.

Checked theforwarder again:

C:\Program Files\SplunkUniversalForwarder\bin>splunk list forward-server
Active forwards:
        localhost:9997
Configured but inactive forwards:
        None

also checked on the indexer splunkd.log:

07-31-2018 10:03:34.311 +0200 INFO  TcpOutputProc - Connected to idx=127.0.0.1:9997, pset=0, reuse=0.
07-31-2018 10:03:40.892 +0200 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-winevtlog.exe"" splunk-winevtlog - WinEventMon::configure: Failed to find Event Log with channel name='Forwarded Events'

on forwarder:

C:\Program Files\SplunkUniversalForwarder\bin>splunk show default-hostname
    Default hostname for data inputs: xxx.

on receiver GUI:

index=_internal sourcetype=splunkd component=TcpInputConfig OR (host=xxx component=StatusMgr)

gives zero events

0 Karma

dagarwal_splunk
Splunk Employee
Splunk Employee

I think hostname should not be a problem. You can change it using:
https://answers.splunk.com/answers/154999/how-can-i-change-the-default-hostname-in-splunk.html

Can I see your props and transforms.conf file located in etc\apps\splunk_app_infrastructure\default\ ?

ALso, Can you try this search:

| mstats count where host=* AND metric_name=* by index,host,metric_name

0 Karma

danielwysockiar
Explorer

Here you go:
transforms.conf

########### Entity Store #################
[em_entities]
external_type = kvstore
collection = em_entities
fields_list = _key,title,state,dimensions,identifier_dimensions,informational_dimensions,imported_date,updated_date, collectors

[em_collector_configs]
external_type = kvstore
collection = em_collector_configs
fields_list = _key,name,title,source_predicate,title_dimension,identifier_dimensions,informational_dimensions,blacklisted_dimensions,monitoring_frequency,monitoring_lag,monitoring_calculation_window,disabled,vital_metrics

[em_groups]
external_type = kvstore
collection = em_groups
fields_list = _key, name, title, filter

[em_thresholds]
external_type = kvstore
collection = em_thresholds
fields_list = _key, name, type_id, type, metric_name, info_min, info_max, warning_min, warning_max, critical_min, critical_max, email_enabled, email_to, email_when

########### Metrics ######################
[metrics-hostoverride]
DEST_KEY = MetaData:Host
REGEX = host=(\S+)
FORMAT = host::$1

########### Transforms for Windows ######################
[value]
REGEX = .*Value=(\S+).*
FORMAT = _value::$1
WRITE_META = true

# Example: object=PhysicalDisk counter="%_Disk_Write_Time"
# Transform - metric_name::PhysicalDisk.%_Disk_Write_Time
[perfmon_metric_name]
REGEX = .*object=(\S+).*counter=(\S+).*
FORMAT = metric_name::$1.$2 metric_type::$1
WRITE_META = true

[instance]
REGEX = .*instance=(\S+).*
FORMAT = instance::$1
WRITE_META = true

as for the "mstats" command, zero results

Checked also Settings -> indexes to find out if there are any metrics indexes, and there is the only one: "em_metrics" assigned to "splunk_app_infrastructure with 0 event count

0 Karma

dagarwal_splunk
Splunk Employee
Splunk Employee

if you are an existing Splunk customer please file a support case so we can pick up some more details about your environment.

Have you tried these troubleshooting docs ?:
http://docs.splunk.com/Documentation/Splunk/7.1.2/Troubleshooting/AdvancedWindowsTroubleshooting
http://docs.splunk.com/Documentation/Splunk/latest/Troubleshooting/Cantfinddata

0 Karma

danielwysockiar
Explorer

Hi, I'll have to have a closer look on those troubleshootig docs.

0 Karma

dagarwal_splunk
Splunk Employee
Splunk Employee

you can all try to uninstall completely and reinstall Splunk Universal Forwarder (using the powershell script on Add Data page)

0 Karma

dagarwal_splunk
Splunk Employee
Splunk Employee

You ran the install script as an administrator, right?

0 Karma
Get Updates on the Splunk Community!

Sending Metrics to Splunk Enterprise With the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

What's New in Splunk Cloud Platform 9.0.2208?!

Howdy!  We are happy to share the newest updates in Splunk Cloud Platform 9.0.2208! Analysts can benefit ...

Want a chance to win $500 to the Splunk shop? Take our IT Incident Management Survey!

  Top Trends & Best Practices in Incident ManagementSplunk is partnering up with Constellation Research to ...