topic Re: Can I define the sourcetype in the log itself? in Getting Data In

Can I define the sourcetype in the log itself?

daniel333 — Mon, 30 Nov 2015 18:43:01 GMT

I happen to have some control over our java logs. Rather than use transforms.conf/props.conf to create various sourcetypes based on the app that is running. Could I simply log out sourcetype in my logs? I believe Splunk will respect it as a KVP. But I don't know if it will really treat it like a sourcetype?

thanks,

Re: Can I define the sourcetype in the log itself?

richgalloway — Mon, 30 Nov 2015 19:21:14 GMT

Splunk uses sourcetypes to know how to parse logs. Without a sourcetype, how will Splunk know how to parse your log to find the sourcetype? Why needlessly index and store another KVP in your events? I'm not saying it can't be done, just that you're probably not saving yourself anything.

Re: Can I define the sourcetype in the log itself?

lguinn2 — Tue, 01 Dec 2015 07:36:34 GMT

I agree with @richgalloway that you gain little by doing this. If you want to take control of your logs, it would be better to implement a naming convention for the log files and directories. Then you could easily assign the sourcetype at input time.

In particular, if you group log files of the same sourcetype into the same directory, you can make entries in the inputs.conf like this example:

[monitor:///var/log/access_combined/]
sourcetype=access_combined

Assigning the sourcetype in inputs.conf is the most efficient way to set the sourcetype. Another alternative (about as efficient) is to include the sourcetype name in the file name. For example: if you had an Apache log named xyz, you could instead name it xyz_access_combined.log
Then you could make the following entry in props.conf for each sourcetype

[source::*access_combined.log]
sourcetype=access_combined

You can certainly add something to your log files. You could even use transforms.conf to create a transform that assigns the sourcetype to each event based on that value - but this is very inefficient! It will significantly slow the indexing of the data as the sourcetype will have to be extracted and assigned to each event, rather than to the entire file at once.

Re: Can I define the sourcetype in the log itself?

gfuente — Tue, 01 Dec 2015 08:04:21 GMT

Hello

Actually you can do this, by adding this code into your logs:

***SPLUNK*** sourcetype=mysourcetype

Then the forwarder will set the sourcetype dinamically, but as Richard said, you still need (or should) define the way the indexer parses that sourcetype, so you will need to have defined in advance all possible sourcetypes you would want to define dinamically in the log itself.

Regards