How can we make sure the appropriate key-value pairs from different web application logs are indexed in Splunk?

[akhilchhugani]

We have multiple web applications that have different information being recorded to make sure the appropriate information is taken from each log. Do we have to assign each type of log a different indexer in inputs.conf while creating new indexers to get exactly what we need from each log?

[richgalloway · Accepted solution]

Don't confuse indexes and indexers. An index is a repository for your data. Indexers are Splunk instances that accept events/logs and store them in indexes.

You do not need separate indexes for each log type. You should, however, specify a sourcetype for each type of log. Sourcetypes tell Splunk how to process the event, such as what the timestamps look like, how the event should be parsed, etc.

[akhilchhugani]

Where can you tell splunk how to process the information based on the sourcetype this was my initial thought, but I was unsure as to how this would occur.

[richgalloway]

Sourcetypes are defined in the props.conf file. You can also define new sourcetypes while using the Field Extractor.

[akhilchhugani]

So if I wanted to create custom sourcetypes, I could add them to the props.conf and edit the inputs.conf and then use the field extractor to edit what key-value pairs they should take from each of these sourcetypes?

[richgalloway]

Correct. Be sure to modify the config files in the local directory, not default.

[akhilchhugani]

I had one more additional question regarding this, when I use the field extractor, will this have to be done every time another log is recorded or only when I set it up for that source type.

[richgalloway]

Once per sourcetype.