topic Re: Using a splunk add-on for infrastucture for a working universal forwarder and enterprise in All Apps and Add-ons

Using a splunk add-on for infrastucture for a working universal forwarder and enterprise

juliennerocafor — Thu, 16 Apr 2020 03:39:50 GMT

Hello, I'm new with Splunk and still exploring how to use it. I was able to successfully create a Splunk Enterprise and Splunk Universal on two separate linux virtual machines. Now, my goal is to create monitoring metrics for cpu usage, etc. I've installed an App for Infrastructure and an add-on for infrastructure in the Splunk Enterprise VM. When adding entities, I can't install the generated linux command since I have restrictions for firewalls and kaspersky and etc. so I just followed this: https://answers.splunk.com/answers/706010/in-the-splunk-app-for-infrastructure-can-you-use-e.html.

Instead of doing the windows version guide, I followed the one in Linux (https://docs.splunk.com/Documentation/InfraApp/1.2.2/Admin/ManageAgents. I've also added an inputs.conf and outputs.conf in my etc/apps/search/local of my splunk forwarder directory. Although when I restart my UF, there are still no entities in my Splunk Enterprise App. Can you help me with this? Thank you in advance!

Inputs.conf

[perfmon://CPU Load]
counters = % C1 Time;% C2 Time;% Idle Time;% Processor Time;% User Time;% Privileged Time;% Reserved Time;% Interrupt Time
instances = *
interval = 30
object = Processor
index = em_metrics
_meta = os::"Linux"

[perfmon://Physical Disk]
counters = % Disk Read Time;% Disk Write Time
instances = *
interval = 30
object = PhysicalDisk
index = em_metrics
_meta = os::"Linux"

[perfmon://Network Interface]
counters = Bytes Received/sec;Bytes Sent/sec;Packets Received/sec;Packets Sent/sec;Packets Received Errors;Packets Outbound Errors
instances = *
interval = 30
object = Network Interface
index = em_metrics
_meta = os::"Linux"

[perfmon://Available Memory]
counters = Cache Bytes;% Committed Bytes In Use;Page Reads/sec;Pages Input/sec;Pages Output/sec;Committed Bytes;Available Bytes
interval = 30
object = Memory
index = em_metrics
_meta = os::"Linux"

[perfmon://System]
counters = Processor Queue Length;Threads
instances = *
interval = 30
object = System
index = em_metrics
_meta = os::"Linux"

[perfmon://Process]
counters = % Processor Time;% User Time;% Privileged Time
instances = *
interval = 30
object = Process
index = em_metrics
_meta = os::"Linux"

[perfmon://Free Disk Space]
counters = Free Megabytes;% Free Space
instances = *
interval = 30
object = LogicalDisk
index = em_metrics
_meta = os::"Linux"

monitor:///var/log/syslog]
disabled = false
sourcetype = syslog

[monitor:///var/log/daemon.log]
disabled = false
sourcetype = syslog

[monitor:///var/log/auth.log]
disabled = false
sourcetype = syslog

[monitor:///var/log/apache/access.log]
disabled = false
sourcetype = combined_access

[monitor:///var/log/apache/error.log]
disabled = false
sourcetype = combined_access

[monitor:///opt/splunkforwarder/var/log/splunk/*.log]
disabled = false
index = _internal

[monitor:///etc/collectd/collectd.log]
disabled = false
index = _internal

Outputs.conf

[tcpout]
defaultGroup = splunk-app-infra-autolb-group

[tcpout:splunk-app-infra-autolb-group]
disabled = false
server = 192.168.56.110:9997

collectd.conf

#
# Config file for collectd(1).
# Please read collectd.conf(5) for a list of options.
# http://collectd.org/
#

##############################################################################
# Global                                                                     
#
#----------------------------------------------------------------------------#
# Global settings for the daemon.                                            
#
##############################################################################

Hostname    "192.168.56.109"
#FQDNLookup   true
#BaseDir     "/var/lib/collectd"
#PIDFile     "/var/run/collectd.pid"
#PluginDir   "/usr/lib64/collectd"
#TypesDB     "/usr/share/collectd/types.db"

#----------------------------------------------------------------------------#
# When enabled, plugins are loaded automatically with the default options    #
# when an appropriate <Plugin ...> block is encountered.                     
#
# Disabled by default.                                                       
#
#----------------------------------------------------------------------------#
#AutoLoadPlugin false

#----------------------------------------------------------------------------#
# When enabled, internal statistics are collected, using "collectd" as the   #
# plugin name.                                                               
#
# Disabled by default.                                                      
#
#----------------------------------------------------------------------------#
#CollectInternalStats false

#----------------------------------------------------------------------------#
# Interval at which to query values. This may be overwritten on a per-plugin #
# base by using the 'Interval' option of the LoadPlugin block:               
#
#   <LoadPlugin foo>                                                        
#
#       Interval 60                                                          
#
#   </LoadPlugin>                                                            
#
#----------------------------------------------------------------------------#
Interval     60

#MaxReadInterval 86400
#Timeout         2
#ReadThreads     5
#WriteThreads    5

# Limit the size of the write queue. Default is no limit. Setting up a limit is
# recommended for servers handling a high volume of traffic.
#WriteQueueLimitHigh 1000000
#WriteQueueLimitLow   800000

##############################################################################
# Logging                                                                    
#
#----------------------------------------------------------------------------#
# Plugins which provide logging functions should be loaded first, so log     #
# messages generated when loading or configuring other plugins can be        #
# accessed.                                                                 
#
##############################################################################

LoadPlugin syslog
LoadPlugin logfile
<LoadPlugin "write_splunk">
        FlushInterval 10
</LoadPlugin>

##############################################################################
# LoadPlugin section                                                        
#
#----------------------------------------------------------------------------#
# Lines beginning with a single `#' belong to plugins which have been built  #
# but are disabled by default.                                               
#
#                                                                            
#
# Lines beginning with `##' belong to plugins which have not been built due  #
# to missing dependencies or because they have been deactivated explicitly.  #
##############################################################################

#LoadPlugin csv
LoadPlugin cpu
LoadPlugin memory
LoadPlugin df
LoadPlugin load
LoadPlugin disk
LoadPlugin interface

##############################################################################
# Plugin configuration                                                       
#
#----------------------------------------------------------------------------#
# In this section configuration stubs for each plugin are provided. A desc-  #
# ription of those options is available in the collectd.conf(5) manual page. #
##############################################################################

<Plugin logfile>
    LogLevel info
    File "/etc/collectd/collectd.log"
    Timestamp true
    PrintSeverity true
</Plugin>

<Plugin syslog>
    LogLevel info
</Plugin>

<Plugin cpu>
    ReportByCpu false
    ReportByState true
    ValuesPercentage true
</Plugin>

<Plugin memory>
    ValuesAbsolute false
    ValuesPercentage true
</Plugin>

<Plugin df>
    FSType "ext2"
    FSType "ext3"
    FSType "ext4"
    FSType "XFS"
    FSType "rootfs"
    FSType "overlay"
    FSType "hfs"
    FSType "apfs"
    FSType "zfs"
    FSType "ufs"
    ReportByDevice true
    ValuesAbsolute false
    ValuesPercentage true
    IgnoreSelected false
</Plugin>

<Plugin load>
    ReportRelative true
</Plugin>

<Plugin disk>
    Disk ""
    IgnoreSelected true
    UdevNameAttr "DEVNAME"
</Plugin>

<Plugin interface>
    IgnoreSelected true
</Plugin>

<Plugin write_splunk>
           server "192.168.56.110"
           port "8088"
           token "SomeGUIDToken"
           ssl true
           verifyssl false
           owner:admin
</Plugin>

#Update Hostname, <HEC SERVER> & <splunk app server> in collectd.conf file above. Also, you can add dimensions as <Dimension "key:value">  to write_splunk plugin (optional)"

Re: Using a splunk add-on for infrastucture for a working universal forwarder and enterprise

gcusello — Thu, 16 Apr 2020 06:36:27 GMT

Hi @juliennerocafort,
To debug your situation, start trying this:

At first, did you enabled your Splunk Enterprise (SE) to receive logs from Universal Forwarder (UF) on port 9997?
then, check the connection using telnet 192.168.56.110 9997 that I suppose to be the address of the SE;
then check if you're receiving the Splunk UF's internal logs: running the search index=_internal host=UF_hostname ;
then, did you installed the Add-On on the UF?

Ciao.
Giuseppe

Re: Using a splunk add-on for infrastucture for a working universal forwarder and enterprise

juliennerocafor — Thu, 16 Apr 2020 07:00:09 GMT

Hello, gcusello.

Yes, I was already receiving forwarded logs from port 9997 even before I install the add-ons.
I wasn't able to use the telnet command, although I can ping the SE from the UF.
There were results showing on my SE search and filter when I run the command.
I just installed the "Splunk Add-on for Infrastructure" in the SE using the 'Browse more apps' option on the homepage. Is it a different installation on the UF?

Regards,
Rockie

Re: Using a splunk add-on for infrastucture for a working universal forwarder and enterprise

gcusello — Thu, 16 Apr 2020 07:20:26 GMT

Hi @juliennerocafort,
if the telnet command is installed on your UF, you can launch the command telnet 192.168.56.110 9997 and check if it's open the route between UF and SE.
Anyway, if the search I suggested have results, it means that the connection between UF and SE is established.
As described in the documentation, the Add-on must be installed both on UF and SE.
You can install, the Add-on untarring it in $SPLUNK_HOME/etc/apps and restarting Splunk.
Check, after untar and before restart, that in all inputs.conf's stanzas there's disabled=0.

Ciao.
Giuseppe

Re: Using a splunk add-on for infrastucture for a working universal forwarder and enterprise

juliennerocafor — Thu, 16 Apr 2020 09:51:33 GMT

Hello @gcusello,

I just installed the add-on on my UF, verified that the inputs.conf includes disabled=false. Although when I restart my SE, there are still no added entities. What else should I check? Should I also install the app for infrastructure in my UF?

Thanks,
Rockie

Re: Using a splunk add-on for infrastucture for a working universal forwarder and enterprise

gcusello — Thu, 16 Apr 2020 10:51:54 GMT

Hi @juliennerocafort,
No, Apps must be installed on SE, Add-Ons on UFs and sometimes on SE.
Did you restarted Splunk on UF after TA's installation?
Every time you modify something on a system (also UFs) by configuration files (TA installation is one of these cases), you have to restart Splunk on that system not on SE.

Ciao.
Giuseppe

Re: Using a splunk add-on for infrastucture for a working universal forwarder and enterprise

juliennerocafor — Thu, 16 Apr 2020 10:54:34 GMT

Hello, @gcusello,
Yes, I have restarted it already. Although, there's still no entities connected.

Regards,
Rockie

Re: Using a splunk add-on for infrastucture for a working universal forwarder and enterprise

gcusello — Thu, 16 Apr 2020 11:03:48 GMT

which is the user running Splunk on UF? if it isn't root, check grants.
Why did you shared collectd.conf? if you're using UF you don't need it.

Ciao.
Giuseppe

Re: Using a splunk add-on for infrastucture for a working universal forwarder and enterprise

juliennerocafor — Thu, 16 Apr 2020 12:04:21 GMT

I'm using root as the user. Oh, I'll just delete it then. I'll try to re-install a new UF on my local and just add the inputs.conf and outputs.conf.

Re: Using a splunk add-on for infrastucture for a working universal forwarder and enterprise

juliennerocafor — Wed, 30 Sep 2020 04:57:59 GMT

Hello @gcusello ,
I've just reinstalled a new forwarder and a new enterprise.

Again, I was able to receive the logs from the UF.
I was also able to get results with the command that you told me awhile ago: ndex=_internal host=UF_hostname.
I've installed the App for Infrastructure on SE and installed the add-on on both the SE and UF.
I also restarted it after installation.
After that, I've updated the inputs.conf file in my UF's /apps/search/local directory. I also put the outputs.conf file there.

Am I putting the conf files in the proper directory? or should they be in SE's /apps/splunk_app_infrastructure/local directory?

Regards,
Rockie

Re: Using a splunk add-on for infrastucture for a working universal forwarder and enterprise

gcusello — Wed, 30 Sep 2020 05:01:42 GMT

Hi @juliennerocafort,
the inputs.conf to modify should be the one in $SPLUNK_HOME/etc/apps/add-on/local , if you haven't it, copy here the one in $SPLUNK_HOME/etc/apps/add-on/default and modify it.

About outputs.conf, for a test you can put it in $SPLUNK_HOME/etc/system/local, in production it's better to create an add-on (called e.g. TA_Forwarders) containing only two files:

outputs.conf,
deploymentclients.conf.

In this way you can easily manage addressing of Deployment Server and Indexers.

When, you solved the present problems, I suggest to analyze the use of Deployment Server to deploy configurations (add-on) on UFs.

Ciao.
Giuseppe

Re: Using a splunk add-on for infrastucture for a working universal forwarder and enterprise

vliggio — Thu, 16 Apr 2020 14:41:26 GMT

The App for Infrastructure goes on the indexers, and the Add-On for Infrastructure goes on both the indexers and the UF's.

The most important command for debugging Splunk is btool. Learn it early and it will be your friend. Since Splunk combines many different config files together, btool allows you to see what Splunk is actually using for its final config. Try this:

/opt/splunkforwarder/bin/splunk btool inputs list --debug

That will show your actual inputs configuration (on a universal forwarder on a Linux box - substitute the application location as necessary on indexers and if you're using Windows). Unfortunately the configs you posted here don't mean anything because Splunk might be getting configs from other directories which override your settings. Play with the command a bit and you'll see (also read up on Splunk config file precedence here - https://docs.splunk.com/Documentation/Splunk/8.0.3/Admin/Wheretofindtheconfigurationfiles)

You do not want to put your inputs.conf into your search app directory. It'll get very confusing very fast. You should have the add-on directory in your splunkforwarder/etc/apps directory and inside the add-on directory you'll see a default directory with an inputs.conf file. Create a local directory in the same folder that the default directory is, and copy the inputs.conf from the default into the local directory, and edit it.

Re: Using a splunk add-on for infrastucture for a working universal forwarder and enterprise

juliennerocafor — Wed, 30 Sep 2020 04:58:02 GMT

Hello @gcusello ,

Apologies but I'm still kinda confused on the path where I should save the inputs.conf. These are the only directories in my $SPLUNK_HOME/etc/apps:

introspection_generator_addon learned
search
splunk_httpinput
splunk_internal_metrics
Splunk_TA_Infrastructure
SplunkUniversalForwarder

Should I just create an 'add-on' folder in my UF?

Thanks,
Rockie

Re: Using a splunk add-on for infrastucture for a working universal forwarder and enterprise

gcusello — Wed, 30 Sep 2020 05:01:44 GMT

No you're speaking of Splunk_TA_Infrastructure, so this is the add-on where put the inputs.conf.
In other words:
copy inputs.conf from $SPLUNK_HOME/etc/apps/Splunk_TA_Infrastructure/default to $SPLUNK_HOME/etc/apps/Splunk_TA_Infrastructure/local and modify this.

Ciao.
Giuseppe

Re: Using a splunk add-on for infrastucture for a working universal forwarder and enterprise

juliennerocafor — Thu, 16 Apr 2020 15:34:16 GMT

Oh, now I get it. Although, I don't have an existing inputs.conf file in the default directory of my add-on so I just created a new one and edit it. On the other hand, there's an existing outputs.conf file in $SPLUNK_HOME/etc/system/local. When I checked it, it already outputs to the SE.

When I restarted it, it still does not work.

Re: Using a splunk add-on for infrastucture for a working universal forwarder and enterprise

gcusello — Wed, 30 Sep 2020 05:01:50 GMT

where did you take the inputs.conf you shared at the beginning of your question?
I assumed that it was from Splunk_TA_Infrastructure.

I read again the Splunk App for infrastructure installation guides:
https://docs.splunk.com/Documentation/InfraApp/2.0.3/Install/SystemRequirements
https://docs.splunk.com/Documentation/InfraApp/2.0.3/Admin/AddData
https://docs.splunk.com/Documentation/InfraApp/2.0.3/Admin/AddDataLinux

And it's different from the other apps, so you should try to follow the installation instructions.

Ciao.
Giuseppe

Re: Using a splunk add-on for infrastucture for a working universal forwarder and enterprise

juliennerocafor — Thu, 16 Apr 2020 15:58:42 GMT

Hello @vliggio ,
I've used the command that you gave me and it showed me the host that I'm actually accessing. It showed me the hostname instead of the ip. When I tried to ping it, I don't get any response at all.. So I just changed it to the host ip address since I can get a response from it.

Also, I added an inputs.conf file in my local directory of the add-on. Although there's no existing input.conf file in the default directory. Is that okay?

Regards,
Rockie

Re: Using a splunk add-on for infrastucture for a working universal forwarder and enterprise

vliggio — Thu, 16 Apr 2020 16:11:37 GMT

Yes. Splunk combines all the files in all app directories (following the precedence rules I linked you to). That's why btool is so important - you can put multiple inputs.conf files in multiple places and could have conflicting settings, and Splunk has specific rules to determine which one it uses. You can use btool to look at any Splunk configuration - just substitute the config file name (ie, ouputs, inputs, indexes, etc).

As for this App/Add-On combo (I haven't installed this specific release), I agree with gcusello - look at the documentation. It's not like most Splunk apps which have inputs.conf. Read the following page on how to enable date inputs:

https://docs.splunk.com/Documentation/InfraApp/2.0.3/Admin/AddData

Also, one minor correction, the Add-On should also be installed on the indexers (in conjunction with the App) - both are needed for the App to function correctly.

Re: Using a splunk add-on for infrastucture for a working universal forwarder and enterprise

dagarwal_splunk — Thu, 16 Apr 2020 19:29:10 GMT

hi,

You are mixing up windows and linux data collection.

"perfmon" inputs in UF is only for Windows metrics.
You need to have "collectd" installed for Linux metrics. Splunk UF only forwards logs for Linux machines. What version of collectd do you have?

Also, you don't need SAI add-on on UF.

Re: Using a splunk add-on for infrastucture for a working universal forwarder and enterprise

vliggio — Thu, 16 Apr 2020 19:33:21 GMT

What do you mean by "Splunk UF only forwards logs for Linux machines"? The UF on Windows certainly collects logs.