All Apps and Add-ons

Using a splunk add-on for infrastucture for a working universal forwarder and enterprise

juliennerocafor
New Member

Hello, I'm new with Splunk and still exploring how to use it. I was able to successfully create a Splunk Enterprise and Splunk Universal on two separate linux virtual machines. Now, my goal is to create monitoring metrics for cpu usage, etc. I've installed an App for Infrastructure and an add-on for infrastructure in the Splunk Enterprise VM. When adding entities, I can't install the generated linux command since I have restrictions for firewalls and kaspersky and etc. so I just followed this: https://answers.splunk.com/answers/706010/in-the-splunk-app-for-infrastructure-can-you-use-e.html.

Instead of doing the windows version guide, I followed the one in Linux (https://docs.splunk.com/Documentation/InfraApp/1.2.2/Admin/ManageAgents. I've also added an inputs.conf and outputs.conf in my etc/apps/search/local of my splunk forwarder directory. Although when I restart my UF, there are still no entities in my Splunk Enterprise App. Can you help me with this? Thank you in advance!

Inputs.conf

[perfmon://CPU Load]
counters = % C1 Time;% C2 Time;% Idle Time;% Processor Time;% User Time;% Privileged Time;% Reserved Time;% Interrupt Time
instances = *
interval = 30
object = Processor
index = em_metrics
_meta = os::"Linux"

[perfmon://Physical Disk]
counters = % Disk Read Time;% Disk Write Time
instances = *
interval = 30
object = PhysicalDisk
index = em_metrics
_meta = os::"Linux"

[perfmon://Network Interface]
counters = Bytes Received/sec;Bytes Sent/sec;Packets Received/sec;Packets Sent/sec;Packets Received Errors;Packets Outbound Errors
instances = *
interval = 30
object = Network Interface
index = em_metrics
_meta = os::"Linux"

[perfmon://Available Memory]
counters = Cache Bytes;% Committed Bytes In Use;Page Reads/sec;Pages Input/sec;Pages Output/sec;Committed Bytes;Available Bytes
interval = 30
object = Memory
index = em_metrics
_meta = os::"Linux"

[perfmon://System]
counters = Processor Queue Length;Threads
instances = *
interval = 30
object = System
index = em_metrics
_meta = os::"Linux"

[perfmon://Process]
counters = % Processor Time;% User Time;% Privileged Time
instances = *
interval = 30
object = Process
index = em_metrics
_meta = os::"Linux"

[perfmon://Free Disk Space]
counters = Free Megabytes;% Free Space
instances = *
interval = 30
object = LogicalDisk
index = em_metrics
_meta = os::"Linux"

monitor:///var/log/syslog]
disabled = false
sourcetype = syslog

[monitor:///var/log/daemon.log]
disabled = false
sourcetype = syslog

[monitor:///var/log/auth.log]
disabled = false
sourcetype = syslog

[monitor:///var/log/apache/access.log]
disabled = false
sourcetype = combined_access

[monitor:///var/log/apache/error.log]
disabled = false
sourcetype = combined_access

[monitor:///opt/splunkforwarder/var/log/splunk/*.log]
disabled = false
index = _internal

[monitor:///etc/collectd/collectd.log]
disabled = false
index = _internal

Outputs.conf

[tcpout]
defaultGroup = splunk-app-infra-autolb-group

[tcpout:splunk-app-infra-autolb-group]
disabled = false
server = 192.168.56.110:9997

collectd.conf

#
# Config file for collectd(1).
# Please read collectd.conf(5) for a list of options.
# http://collectd.org/
#

##############################################################################
# Global                                                                     
#
#----------------------------------------------------------------------------#
# Global settings for the daemon.                                            
#
##############################################################################

Hostname    "192.168.56.109"
#FQDNLookup   true
#BaseDir     "/var/lib/collectd"
#PIDFile     "/var/run/collectd.pid"
#PluginDir   "/usr/lib64/collectd"
#TypesDB     "/usr/share/collectd/types.db"

#----------------------------------------------------------------------------#
# When enabled, plugins are loaded automatically with the default options    #
# when an appropriate <Plugin ...> block is encountered.                     
#
# Disabled by default.                                                       
#
#----------------------------------------------------------------------------#
#AutoLoadPlugin false

#----------------------------------------------------------------------------#
# When enabled, internal statistics are collected, using "collectd" as the   #
# plugin name.                                                               
#
# Disabled by default.                                                      
#
#----------------------------------------------------------------------------#
#CollectInternalStats false

#----------------------------------------------------------------------------#
# Interval at which to query values. This may be overwritten on a per-plugin #
# base by using the 'Interval' option of the LoadPlugin block:               
#
#   <LoadPlugin foo>                                                        
#
#       Interval 60                                                          
#
#   </LoadPlugin>                                                            
#
#----------------------------------------------------------------------------#
Interval     60

#MaxReadInterval 86400
#Timeout         2
#ReadThreads     5
#WriteThreads    5

# Limit the size of the write queue. Default is no limit. Setting up a limit is
# recommended for servers handling a high volume of traffic.
#WriteQueueLimitHigh 1000000
#WriteQueueLimitLow   800000

##############################################################################
# Logging                                                                    
#
#----------------------------------------------------------------------------#
# Plugins which provide logging functions should be loaded first, so log     #
# messages generated when loading or configuring other plugins can be        #
# accessed.                                                                 
#
##############################################################################

LoadPlugin syslog
LoadPlugin logfile
<LoadPlugin "write_splunk">
        FlushInterval 10
</LoadPlugin>

##############################################################################
# LoadPlugin section                                                        
#
#----------------------------------------------------------------------------#
# Lines beginning with a single `#' belong to plugins which have been built  #
# but are disabled by default.                                               
#
#                                                                            
#
# Lines beginning with `##' belong to plugins which have not been built due  #
# to missing dependencies or because they have been deactivated explicitly.  #
##############################################################################

#LoadPlugin csv
LoadPlugin cpu
LoadPlugin memory
LoadPlugin df
LoadPlugin load
LoadPlugin disk
LoadPlugin interface

##############################################################################
# Plugin configuration                                                       
#
#----------------------------------------------------------------------------#
# In this section configuration stubs for each plugin are provided. A desc-  #
# ription of those options is available in the collectd.conf(5) manual page. #
##############################################################################

<Plugin logfile>
    LogLevel info
    File "/etc/collectd/collectd.log"
    Timestamp true
    PrintSeverity true
</Plugin>

<Plugin syslog>
    LogLevel info
</Plugin>

<Plugin cpu>
    ReportByCpu false
    ReportByState true
    ValuesPercentage true
</Plugin>

<Plugin memory>
    ValuesAbsolute false
    ValuesPercentage true
</Plugin>

<Plugin df>
    FSType "ext2"
    FSType "ext3"
    FSType "ext4"
    FSType "XFS"
    FSType "rootfs"
    FSType "overlay"
    FSType "hfs"
    FSType "apfs"
    FSType "zfs"
    FSType "ufs"
    ReportByDevice true
    ValuesAbsolute false
    ValuesPercentage true
    IgnoreSelected false
</Plugin>

<Plugin load>
    ReportRelative true
</Plugin>

<Plugin disk>
    Disk ""
    IgnoreSelected true
    UdevNameAttr "DEVNAME"
</Plugin>

<Plugin interface>
    IgnoreSelected true
</Plugin>

<Plugin write_splunk>
           server "192.168.56.110"
           port "8088"
           token "SomeGUIDToken"
           ssl true
           verifyssl false
           owner:admin
</Plugin>

#Update Hostname, <HEC SERVER> & <splunk app server> in collectd.conf file above. Also, you can add dimensions as <Dimension "key:value">  to write_splunk plugin (optional)" 
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @juliennerocafort,
the inputs.conf to modify should be the one in $SPLUNK_HOME/etc/apps/add-on/local , if you haven't it, copy here the one in $SPLUNK_HOME/etc/apps/add-on/default and modify it.

About outputs.conf, for a test you can put it in $SPLUNK_HOME/etc/system/local, in production it's better to create an add-on (called e.g. TA_Forwarders) containing only two files:

  • outputs.conf,
  • deploymentclients.conf.

In this way you can easily manage addressing of Deployment Server and Indexers.

When, you solved the present problems, I suggest to analyze the use of Deployment Server to deploy configurations (add-on) on UFs.

Ciao.
Giuseppe

0 Karma

juliennerocafor
New Member

Hello @gcusello ,

Apologies but I'm still kinda confused on the path where I should save the inputs.conf. These are the only directories in my $SPLUNK_HOME/etc/apps:

introspection_generator_addon learned
search
splunk_httpinput
splunk_internal_metrics
Splunk_TA_Infrastructure
SplunkUniversalForwarder

Should I just create an 'add-on' folder in my UF?

Thanks,
Rockie

0 Karma

gcusello
SplunkTrust
SplunkTrust

No you're speaking of Splunk_TA_Infrastructure, so this is the add-on where put the inputs.conf.
In other words:
copy inputs.conf from $SPLUNK_HOME/etc/apps/Splunk_TA_Infrastructure/default to $SPLUNK_HOME/etc/apps/Splunk_TA_Infrastructure/local and modify this.

Ciao.
Giuseppe

0 Karma

juliennerocafor
New Member

Oh, now I get it. Although, I don't have an existing inputs.conf file in the default directory of my add-on so I just created a new one and edit it. On the other hand, there's an existing outputs.conf file in $SPLUNK_HOME/etc/system/local. When I checked it, it already outputs to the SE.

When I restarted it, it still does not work.

0 Karma

gcusello
SplunkTrust
SplunkTrust

where did you take the inputs.conf you shared at the beginning of your question?
I assumed that it was from Splunk_TA_Infrastructure.

I read again the Splunk App for infrastructure installation guides:
https://docs.splunk.com/Documentation/InfraApp/2.0.3/Install/SystemRequirements
https://docs.splunk.com/Documentation/InfraApp/2.0.3/Admin/AddData
https://docs.splunk.com/Documentation/InfraApp/2.0.3/Admin/AddDataLinux

And it's different from the other apps, so you should try to follow the installation instructions.

Ciao.
Giuseppe

0 Karma