Question about Inputs.conf

[zekiramhi]

Hello,

I have made a new app under deployment apps with the following inputs.conf

[monitor:///root/something/something/something/something/]
index = test
whitelist=console-202[\S\s]+\.log$

whitelist is written to input filenames such as console-2020-06-02.log etc

 I have not created any sourcetype for the index, so I do not have a props.conf file on the deployment app, neither on the searchheads. I have reloaded the server class that is linked to the host and app but I do not see any attempts to monitor the path I have given on the following spl query:

"index=_internal sourcetype=splunkd *something*"

Am I missing something on the inputs.conf? Am I forced to put a sourcetype? Cant I create my own custom sourcetpe via the gui or do I have to create a props and transforms conf for a sourcetype that does not exist?

Any help is appreciated,

Regards,

[richgalloway]

Every input should have a sourcetype associated with it and every sourcetype should have a props.conf stanza.  This keeps Splunk from having to guess about how to ingest your data and possibly getting it wrong.  You can create a sourcetype in the UI at Settings->Source types->New Source Type.

When you created the new app did you specify the Restart Splunkd option?  If not, then the inputs.conf has not taken effect.

[zekiramhi]

Maybe because I have never created a sourcetype for this index, is the reason it is not accepting to monitor this path. My main goal was to have Splunk ingest the data into Splunk and then create a Sourcetype for the incoming log on the index via the gui.

How I go about doing my sourcetype is the following:

1. Place the log sample on a test server via Add Data > Upload 

2. Check if any of the pretrained sourcetypes produce a healthy result, which in this case they havent. So I proceeded to write my own regex for the key value pairs

3. Here lies the main question, now should I just copy the Avanced Settings "Copy to Clippboard" which I am pleased with how it has extracted the time and split the events as I want but the thing is it has set the sourcetype as [ __auto__learned__ ]  which I dont think I should change for the events to extract the time automatically.

So now do I create a props.conf with  [ __auto__learned__ ] and then reload the serverclass for the logs to flow? (if I go this path do I name the sourcetype to : [ __auto__learned__ ] in inputs.conf?) or can I just set the sourcetype to some dummy name in inputs.conf that does not exist in which I create via gui after the log arrives?

Apologies for the long explanation, but hopefully I have made myself clear

Regards,

[richgalloway · Accepted solution]

Once you have used the Add Data wizard to process a sample data file and are happy with the results, click the Save As button to save your settings as a new sourcetype with a name you specify.  Put that same name in the inputs.conf file.

[zekiramhi]

Hello,

I have done as you said, and do see the logs that I want being ingested via

INFO LicenseUsage - type=Usage.. Logs,

But I do not see the logs when I try to search for the index or sourcetype, is there anything I am supposed to check?

[richgalloway]

Re-run your INFO LicenseUsage - type=Usage.. Logs search in Verbose Mode.  Check the index and sourcetype fields in the events returned.  Use those values when you search by index or sourcetype.

[zekiramhi]

It was still in the process of injesting, after checking for all time I was able to see my logs. Many Thanks 😁

[rnowitzki]

Hi @zekiramhi,

Is the user that runs Splunk (I guess "splunk") able to read the files in the monitoring stanza?

Sourcetype is not mandatory (but recommended). Per Documentation: 
"If not set, the indexer analyzes the data and chooses a source type."

BR
Ralph

[zekiramhi]

Hello,

Yes, I have given specific rights for the responsible user just as I have with my previous deployment app which is working.

Thanks,

[rnowitzki]

Is the app in the serverclass configured to restart the forwarder?

(just checking the easy/obvious stuff 🙂 )

[zekiramhi]

Yes, I have forgotten to do so but I have applied and reloaded the serverclass with no changes unfortunately 😕

Thank you for the suggestion 😁