topic Re: Getting hostname from combined logfile in Getting Data In

Getting hostname from combined logfile

drugscom — Wed, 15 Aug 2012 02:49:59 GMT

Our logs are combined on our logserver with scribe and they look like:

[web1] Time: 120807  0:08:21
[web1] Something something
[web1] Something else
[web4] Time: 120807  0:08:25
...

How can I strip the [web1] from each line and use that as the hostname in Splunk?

Thanks

Re: Getting hostname from combined logfile

mwhite_splunk — Wed, 15 Aug 2012 03:30:22 GMT

You can tell Splunk to extract the field with the interactive field extractor, doc here:

http://docs.splunk.com/Documentation/Splunk/latest/User/InteractiveFieldExtractionExample

To combine the events, you'll have to use the transaction function within Splunk during a search. Here:

http://blogs.splunk.com/2010/09/01/event-correlation/

is a Splunk blog describing how to achieve this. Let us know if this is not what you are looking for.

Re: Getting hostname from combined logfile

lguinn2 — Wed, 15 Aug 2012 05:20:55 GMT

Because host is one of the few indexed fields, rather than search-time fields, you will have to do things a little differently. You should not use the Interactive Field Extractor or any other search-time method for creating the host field.

In props.conf, put

[yoursourcetype]
SHOULD_LINEMERGE=false
TIME_PREFIX=Time: 
TIME_FORMAT = %y%m%d %H:%M:%S
TRANSFORMS-my-host = extract-my-host

In transforms.conf, put

[extract-my-host]
DEST_KEY = MetaData:Host
REGEX = ^\[(\S+?)]
FORMAT = host::$1

I don't know what sourcetype you gave this data, but you will need to substitute that for yoursourcetype in props.conf. I also threw in a few more settings that will speed up Splunk's parsing of the input stream, and make sure the timestamp is properly interpreted. I assume that this log contains only single-line events.

Also, the REGEX assumes that the host name always appears at the beginning of each line, enclosed in square brackets.

Let us know if that doesn't work.