topic Re: sourcetype best practices in Getting Data In

sourcetype best practices

a212830 — Fri, 10 Aug 2012 15:11:29 GMT

Hi,

I'm looking for some help on sourcetype naming. I have a bunch of logfiles - some apache error logs, some apache access logs, some custom application error logs. I want to give my customers an easy way to search on these logs (there will be dozens of them). Should I use a pretrained source type? Wouldn't that make it more difficult to search on the logs? If I use a custom sourcetype (say "appname"), will Splunk recognize the logfile formats?

Re: sourcetype best practices

hexx — Mon, 28 Sep 2020 12:14:56 GMT

The concept of sourcetype was introduced so that a metadata field associated with an event would describe the nature of the data, which typically tells us something about the structure of the data rather than its precise origin. "Where is this data coming from?" is a question best answered with the 'host' and 'source' metadata fields. The sourcetype is rather there to answer "What kind of data is this?".

For that reason, I would not recommend to assign the same sourcetype to access logs and application logs, for example. You are probably better off using a pre-trained sourcetype whenever one is available, such as 'access_common' or 'access_combined' for HTTPD access logs. This will bring the benefit of pre-packaged field extractions, among other things.

Note that most pre-trained sourcetypes are defined in $SPLUNK_HOME/etc/system/default/props.conf.

Re: sourcetype best practices

a212830 — Mon, 13 Aug 2012 18:00:10 GMT

Thanks, this helps me understand the usage better. Still, if I have dozens of logfiles, across multiple hosts, and I want to search them, how would I easily do that? I don't want to type in each host or logfile - that's a lot of work. Is there an alias, or something like that?

Re: sourcetype best practices

hexx — Mon, 13 Aug 2012 20:12:57 GMT

There are many ways to do this, and it really depends on what qualifies the event set that you want your search to return. You can use:

- Wildcards in your search terms
- Eventtypes
- Tags

Re: sourcetype best practices

ChrisG — Mon, 13 Aug 2012 21:03:14 GMT

I do also recommend reading http://docs.splunk.com/Documentation/Splunk/latest/Data/Whysourcetypesmatter and the topics that follow it.