topic Sourcetype would increment? in Getting Data In

Sourcetype would increment?

hochit — Tue, 28 Dec 2010 12:36:05 GMT

Hi,

I found the sourcetype that I've been set was growing with incremental digit. e.g. sourcetype=abc-1, sourcetype=abc-2, sourcetype=abc-3 ...

Do you know why this happens?

Thanks! Philip

Re: Sourcetype would increment?

Simeon — Wed, 29 Dec 2010 00:36:20 GMT

Splunk will try to learn your sourcetype automatically. As a result, similar source files with varying event punctuation may show up with the same name with incremental digits.

The correct way to fix this is to explicitly set the sourcetype for your data source. This can be done within inputs.conf or via the gui. In some scenarios, you may not be able to explicitly set this as the input source has many sourcetypes. In that case, you must use a props.conf setting to identify individual sourcetypes (somewhat of an advanced topic).

If you want to get rid of the automatic sourcetyping, you can disable the learned sourcetypes although that is not recommended. To do this, you can simply disable the learned app via the UI. Unfortunately, you cannot go back and change the indexed sourcetype value without re-indexing.

Re: Sourcetype would increment?

rotten — Wed, 29 Dec 2010 05:14:03 GMT

It isn't so obvious how to force a sourcetype using props.conf. Here is how we do it:

In props.conf on the indexer you need to identify the files you want to map to a sourcetype using some sort of expression that looks similar to the inputs.conf expressions:

[source::/my/appserver/path/.../logs/access*]
TRANSFORMS-appserver_access = fix_appserver_access_sourcetype

Then you need to put this in transforms.conf:

[fix_appserver_access_sourcetype]
REGEX = .
FORMAT = sourcetype::appserver_access
DEST_KEY = MetaData:Sourcetype

Then you need to restart the indexer. All new files that are identified will get the new sourcetype. The files it has already seen will still get mapped with the old sourcetype.

Another note: If you want to force the sourcetype on the inputs.conf you do that on the forwarder.

You can also tag sources that have already been indexed to give them an sourcetype alias. This can get really tedious if you have hundreds and hundreds of files that were indexed with the wrong sourcetype.