Can I create indexes.conf and inputs.conf files on my search heads to send /var/log/ logs to my indexer cluster?

[halbeisendv]

My SHC of 3 members is Linux. I need to create an inputs.conf to ingest /var/log/* and send them to my indexer-cluster. _internal data

from all of my servers is being indexed properly so I believe that the data flow is correct. I believe I need to do two things: 1)

create an indexes.conf file on each search head and 2) create an inputs.conf file on each search head.

Step 1) On my deployer, I created /opt/splunk/etc/master-apps/_cluster/local/indexes.conf and executed splunk apply shcluster-bundle

without errrors. This is the contents of indexes.conf.

[linux]
coldPath = $SPLUNK_DB/linux/colddb
enableDataIntegrityControl = 0
enableTsidxReduction = 0
homePath = $SPLUNK_DB/linux/db
maxTotalDataSizeMB = 512000
thawedPath = $SPLUNK_DB/linux/thaweddb

I cannot find the indexes.conf file on any of my search heads.

2) I also created /shcluster/apps/locallinux/local/inputs.conf and executed splunk apply shcluter-bundle without errors. This is the contents of inputs.conf.

[monitor:///var/log/messages]
disabled = false
index = linux
sourcetype = syslog

[monitor:///var/log/cron]
disabled = false
index = linux
sourcetype = syslog

Same problem as above, I cannot find the inputs.conf file on any of my search heads.

In a separate, but bigger picture of what I am trying to accomplish, on my License Server and on my Monitoring server, I created a linux index and used the web gui to create the inputs AND I have SPLUNK_HOME/etc/system/local/outputs.conf as below.

[indexAndForward]
index = false

[tcpout]
defaultGroup = DSCA_Indexers
forwardedindex.filter.disable = true
indexAndForward = false

[tcpout:DSCA_Indexers]
server=10.20.38.11:9997, 10.20.38.12:9997, 10.20.38.13:9997

My linux information gets to the indexers.

The desired goal is to send ALL Enterprise Server Linux /var/log/* to the indexers.

[ddrillic]

-- Step 1) On my deployer, I created /opt/splunk/etc/master-apps/_cluster/local/indexes.conf

It's the replication server.

About -

[monitor:///var/log/messages]
disabled = false
index = linux
sourcetype = syslog

[monitor:///var/log/cron]
disabled = false
index = linux
sourcetype = syslog

Better create a unique sourcetype for each one.

-- 2) I also created /shcluster/apps/locallinux/local/inputs.conf and executed splunk apply shcluter-bundle without errors.

This is done on the deployment server at $SPLUNK_HOME/etc/deployment-apps/locallinux/local/inputs.conf and it works in conjuction with $SPLUNK_HOME/etc/system/local/serverclass.conf.

[halbeisendv]

I am not sure of what you are trying to communicate by writing, "it's the replication server." My understanding is that configuration of search head members in a cluster is done by pushing configuration changes from the DEPLOYER, not a DEPLOYMENT Server. Please clarify.

[ddrillic]

/opt/splunk/etc/master-apps/_cluster/local/indexes.conf lives on the replication server ; -)

[gcusello · Accepted solution]

Hi halbeisendv,
you don't need to create an index on your SHs, you need only to forward SHs logs to indexers where you prevently created the correct index.
I usually configure all my Splunk servers (SH, DS, Deployer, DMC, ...) to forward internal logs to indexers so I can monitor all Splunk Infrastructure.
So inputs.conf and outputs.conf are correct, indexes.conf doesn't serve,

Bye.
Giuseppe

[halbeisendv]

Giuseppe -
Do you mean that the inputs.conf file for each search head member in the cluster should be created at SH{1-3}:/$SPLUNK_HOME/etc/system/local/inputs.conf?

[gcusello]

Hi halbeisendv,
NO! It isn't a best practice to put inputs.conf in $SPLUNK_HOME/etc/system/local/, it's better to put it always in a dedicated TA or App to deploy.

In my previous answer, I was meaning that you don't need to have an indexes.conf on Search Heads to send logs to your Indexers, you have only to configure forwarderding on your SHs.

Anyway, to deploy inputs.conf, I suggest to prepare a new app (containing at least your inputs.conf) and deploy it using deployer ( http://docs.splunk.com/Documentation/Splunk/7.0.0/DistSearch/PropagateSHCconfigurationchanges ).

Think to use "Splunk Add-on for Unix and Linux" ( https://splunkbase.splunk.com/app/833/ ) where you can find the paths you want to monitor and other useful things to monitor *nix systems.

Bye.
Giuseppe

[halbeisendv]

Got it! Thanks for your input.