How to create a new index?

[sherrysafdar]

I am collecting the log files from my syslog server and defined the index for the source path but it is still sending the the events to the main index.

Need to change the index for the event.

Please help!

[woodcock]

On your indexer you need to create an index using the indexes.conf file:
https://docs.splunk.com/Documentation/Splunk/latest/Admin/Indexesconf

Once the index is defined there, you need to reference it inside of the inputs.conf on your syslog server like this:

[monitor://.......]
index=YourIndexNameHere

[sherrysafdar]

I am unable to find the indexes.conf under /opt/splunk/etc/system/local I wonder if I need to need to create one? if that doesn't exist?

Also, on my syslog my current inputs.conf looks like below with the default installation.

[default]
host = SP-FWDR

Do I need to reference each index inside the inputs.conf whatever I need to be in a separate indexes? how about the rest of the events will it go to the main?

[woodcock]

No, do not put your stuff in /opt/splunk/etc/system/local. Create your own app on your indexers in /opt/splunk/etc/apps/YourCompany_all_indexes/default/indexes.conf. Yes, you reference that index name inside of inputs.conf on your syslog server. If you do not specify any index name, then by default, your events will go to main. If you specify and index name/value that does not exist, then the events will go nowhere and get dropped (logging Received event for unconfigured/disabled/deleted in index=_internal ), unless you have defined a lastChanceIndex on your indexers.

[ddrillic]

Sorry for jumping in @woodcock - we put our indexes under $SPLUNK_HOME/etc/apps/YourCompany_all_indexes/local/indexes.conf (local versus your recommendation of default).

In the past, on an older platform, we placed the indexes under - $SPLUNK_HOME/etc/master-apps/_cluster/local/indexes.conf

So, we see three variations -

1) $SPLUNK_HOME/etc/apps/YourCompany_all_indexes/default/indexes.conf
2) $SPLUNK_HOME/etc/apps/YourCompany_all_indexes/local/indexes.conf
3) $SPLUNK_HOME/etc/master-apps/_cluster/local/indexes.conf

Why is #1 the preferred way?

[woodcock]

The person who authors the app should deploy his configurations in default. That way if the person who is using the app needs to updated/override/configure the settings to suit his situation, he can deploy his configurations in local to override the author's defaults. Splunk PS has the bad habit of deploying their configurations in custom_app/local because it suits them better to have configurations that the user cannot override because they prefer things to stay the way that they think they should be. I think that this is a mistake. In any case EVERYBODY agrees that nothing should be deployed in $SPLUNK_HOME/etc/system/.

[ddrillic]

Much appreciated @woodcock , what about the $SPLUNK_HOME/etc/master-apps/_cluster/local/indexes.conf location ?

[woodcock]

That directory has the highest precedence of all, even higher than $SPLUNK_HOME/etc/system/local/, so is for emergencies only.

[ddrillic]

oh oh wow - much appreciated @woodcock !!!

[codebuilder]

If you deploy indexes.conf via the master at /opt/splunk/etc/master-apps/_cluster/local/indexes.conf then your indexes.conf on the peer nodes (indexers) will reside at /opt/splunk/etc/slave-apps/_cluster/local/indexes.conf, not at /etc/system/local.

In a clustered environment, I'm not a fan of deploying indexes.conf any other way, though it can be done (e.g. in a custom app).
If that app gets deleted, or you inadvertently deploy with the app directory missing, or wrong permissions, etc. then you're going to have problems.

The /opt/splunk/etc/master-apps/_cluster/local/ is a special directory intended just for this purpose, and is documented as such:
https://docs.splunk.com/Documentation/Splunk/7.0.4/Indexer/Updatepeerconfigurations