Can anyone explain me how to on board data.

[Rocky31]

I was hired in an organization as a Splunk onboard specialist, I don't know much about onboarding data. I had gone through getting data in docs but that is not helpful to deal in real time.

Our environment 325 GB/ per day
7 indexers, 4 SH, 100 UF.

Can anyone please share your onboarding knowledge with me.

splunk learner.
Rocky.

[gcusello · Accepted solution]

Hi Rocky31,
at first I suggest to quickly start with Splunk free training ( https://www.splunk.com/view/SP-CAAAPX9 and https://www.splunk.com/view/SP-CAAAHSM ) and then partecipate to a splunk certification plan, at least as Administrator, better as Architect.

Anyway at first you should define your perimeter and monitoring needs (logs files, scripts, wineventlogs, etc... to take for monitoring) to understand and configure your architecture.
In this way you know which logs are you waiting for and how to prepare your Splunk distributed architecture,

Then you have to define which service level you need, in other words do you need clustered indexers and/or clustered search heads?
So you'll have a configured Splunk distributed search architetcture that can index and search logs.

Then if you have Forwarders, you need to use a dedicated Deployment Server to deploy configurations to Forwarders (for more than 50 forwarders must be a dedicated server).
Configurations are in apps called Technical Addons (TAs) that contain information about the indexers to send data (outputs.conf) and objects to monitor (files, scripts, wineventlogs, etc...).
Remember that Deployment Server is the only configuration that must be done locally on forwarders.

When you have clear ideas about monitoring requirements you can start to prepare your TAs:
at first configure a TA containing only outputs.conf to correctly address your Forwarders to send logs to the Indexers.

You can check the connecting Forwarders runnning a simple search on Search Heads (index=_internal | stats count by host) and verify if all your Forwarders are connected.

Then prepare your TAs to ingest data for the required monitoring.
When you're sure to index the correct data you can start to prepare your searches to display the situations to monitor (errors, health status, etc...).

I hope to be useful for you, anyway first thing is training, but in addition to read answers.splunk.com is a good idea to understand behaviour.

Bye.
Giuseppe

[Rocky31]

I have another question, why i don't find output.conf file in splunkforwarder in free splunk on my local instance

location:

MacBook-Pro:local RRRR$ pwd
/Applications/splunkforwarder/etc/system/local

[gcusello]

Hi Rocky31,
Because outputs.conf is created when you run the following command
./splunk add forward-server :
see http://docs.splunk.com/Documentation/Forwarder/7.0.0/Forwarder/Configuretheuniversalforwarder

I suggest to put outputs.con in a dedicated TA to manage using a Deployment Server not in $SPLUNK_HOME/etc/system/local

Bye.
Giuseppe

[Rocky31]

Appreciate it sir for you taking time to look in to.