Can't get sourcetype right

[Branden]

Hi.

Seems like a lot of people have a question similar to this, but maybe I am missing something simple. I'm monitoring logs in a directory (/foo/barr/logs/APPSRV*). Logs are sent to the indexer via the SplunkLightForwarder. The wild card is because the log file names are dynamic (they have a timestamp on the end of them)

I created a props.conf in /splunk/etc/apps/search/local with the following entry on the SplunkLightForwarder:

[source:///foo/barr/logs/APPSRV*]
sourcetype=appsrv

Unfortunately, the sourcetype still comes out wrong. It shows up as APPSRV-too_small. I then tried putting props.conf in /splunk/etc/apps/search/default, but that didn't help either.

I think I'm missing something fundamental here...

Thanks in advance for your help.

[rotten]

Splunk support showed us how to do it using an approach like this:

---

inputs.conf on the lightweight forwarder:

[monitor:///foo/bar/logs/]
disabled = false
host = myServer_myApplication
crcSalt = <SOURCE>
blacklist = \.(tar|gz|bz2)

props.conf on the indexer:

[source::/foo/bar/logs/.../*]
TRANSFORMS-foobarlogs = fix_foo_bar_logs_sourcetype

transforms.conf on the indexer:

[fix_foo_bar_logs_sourcetype]
REGEX=.
FORMAT=sourcetype::foo_bar_log
DEST_KEY=MetaData:Sourcetype

---

This approach is the only way we have found to reliably set the sourcetype for the vast majority of our logs.

If we pick up the logs on the indexer, we can simplify this by setting the sourcetype in the inputs.conf directly:

[monitor:///foo/bar/logs/]
disabled = false
sourcetype = foo_bar_log
host = myServer_myApplication
crcSalt = <SOURCE>
blacklist = \.(tar|gz|bz2)

[BunnyHop]

If you starting seeing "**-too_small" it means that the indexing is being learned, and not using your props.conf or transforms.conf settings. So, a couple things:

First, you shouldn't replace any of the documents/configuration files within the Default folder. Use either the Apps folder or the Local folder for your custom config. Second, there are a few bugs in the past (true with IIS at some point) that no matter how you specify the sourcetype, it will only grab the name, and learn the rest automatically. So that's something that needs to be fixed on the software. Third, when you re-add/re-index, make sure you clear the learned configurations. This is located on the Apps folder under learned.

[Stephen_Sorkin]

Your first approach didn't work because of the difference in naming convention for stanzas in inputs.conf and props.conf.

In inputs.conf, stanzas look like URIs, so they have a scheme (say monitor, tcp or udp) followed by "://" followed by a path.

On the other hand, props.conf just looks at the fields source, sourcetype or host. So the stanzas in props.conf should be named: [<sourcetype>] [source::<source>] [host::<host>]

The "::" here is a inherited from when all fields in Splunk were indicated by field::value rather than field=value.

[Lowell]

Your first approach should have worked, but I image you had a typo or permissions setting getting in the way. I would also like to point out that there are limitations in simply forcing a sourcetype on all of your inputs.conf stanzas as was suggested above. For example, say you had two different types of logs in /foo/barr/logs/APPSRV* and you want to use two different sourcetypes? How would you get the proper sourcetype assigned? (Splunk 4.1 makes this slightly easier, but there are still some limitations.)

I highly recommend using the following command line tool to debug your .props files:

splunk test sourcetype /foo/barr/logs/APPSRV/some_log_file.log

This should return the sourcetype associated with your log file and show you all the props settings associated with it.

---

I put together a list of common debugging techniques on a different post. You may find some of it helpful:

What’s the best way to track down props.conf problems?

[treinke · Accepted solution]

In $SPLUNK_HOME/etc/system/local/inputs.conf

If you are using the file system change:

[fschange:/foo/barr/logs/APPSRV*]
sourcetype = appsrv

For monitoring the files/folders:

[monitor:///foo/barr/logs/APPSRV*]
sourcetype = appsrv

A great resource to find out more about the inputs.conf is: http://www.splunk.com/base/Documentation/4.1.4/admin/Inputsconf

[Branden]

Worked great, thanks!