Teaching Splunk the fields in a custom log format

[dpadams]

I'm new to Splunk and may have a question that's a bit out of my depth. I've got Splunk configured now to aggregate a collection of standard Web logs from a group of servers. The next step I'd like to take is to have Splunk integrate a custom log format from our server app. The log format is entirely up to us but I figure it should include:

-- Timestamp for Splunk -- Host name/address for Splunk

Beyond that, we'll want several custom fields like an error name, diagnostic text and the request URL for a Web-provoked error.

What I'm trying to sort out is how to teach Splunk what my fields mean. I've seen mention of transform.conf files but don't follow how this would work. I've seen a few other questions but didn't quite nail the answer.

It seems like this out to be a standard, if not universal, thing to do with Splunk. Can anyone point me in the right direction?

Thanks very much for any help.

[lguinn2 · Accepted solution]

There are several ways to accomplish this, but the easiest way is to create your log with a format similar to

timestamp host=hostname errorName=xxxx errorText="your message here" requestURL=http://page.site.com/path/page.html (and so on)

Splunk can automatically process most common timestamp formats, so just pick something sensible. Your timestamp should definitely include a timezone designation.

For the remainder of the log format, use the form name=value and Splunk will automatically identify your fields for you. BTW, it is not necessary for every log entry to contain exactly the same fields. Put quotes around values that contain whitespace (like the error text in the example).

In my example, I used camel case for the field names, but you can name the fields whatever you like -- using letters, numbers and underscores. The field name must begin with a letter. So use error_text instead of errorText if you prefer. BTW, field names are case-sensitive: errortext and errorText are not the same thing.

If you do it this way, your log will contain more characters. If you want to go with a more compact log format, you will need to define field extractions.

[dpadams]

Thanks very much for the help, it's much appreciated. The details and key terms you provided give me a good path forward. I'll be digging into the field extractions features and docs next. -- Thanks