topic Re: Defining fields in transforms.conf in Getting Data In

Defining fields in transforms.conf

kmattern — Wed, 09 Nov 2011 16:07:35 GMT

Splunk Version 4.0.11

I’m working on getting Splunk to consume “log” files that I have dumped from a SQL Server helpdesk database. There are a lot of date and time fields in the events as well as dates and times that are included in the free-form text fields. I have managed to extract fields for my journal logs. My assignment log looks like this



2011-10-03^09:15:37^ ^ ^00021361^Brandon^ ^ ^

2011-10-03^08:10:35^2011-10-03^14:51:43^00021584^Bonnie^Completed^15^

2011-10-03^08:25:28^2011-10-04^07:17:13^00021585^Bonnie^Completed^20^

2011-10-03^08:40:19^2011-10-05^13:40:38^00021587^Bonnie^Completed^30^

2011-10-03^08:59:21^2011-10-04^07:15:03^00021588^Bonnie^Completed^20^

2011-10-03^09:14:46^2011-10-03^14:05:20^00021589^Patrick^Completed^15^

2011-10-03^09:24:33^2011-10-04^07:02:54^00021591^Stephanie^Completed^10^

2011-10-03^10:21:26^2011-10-03^14:25:04^00021592^Jeff^Completed^15^

2011-10-03^10:42:09^2011-10-03^10:42:25^00021593^Robert^Completed^10^

transforms.conf looks like this



[AsgLog] 

DELIMS="^" 

FIELDS="DateAssign",TimeAssign","DateResolv","TimeResolv","CallID","Assignee","Resolution","TotalAsgnmntTime"

Splunk consumes the data and I can search it but the fields defined in transforms.conf are not available. I’ve tried extracting the fields but with two date fields and two time fields Splunk doesn’t seem able to discriminate between them.
What am I doing wrong?

Re: Defining fields in transforms.conf

Drainy — Wed, 09 Nov 2011 16:55:26 GMT

Have you tried it with the FIELDS not surrounded by " "'s. E.g.

FIELDS=DateAssign,TimeAssign,DateResolv,TimeResolv,CallID,Assignee,Resolution,TotalAsgnmntTime

Also, have you restarted Splunk? this is required to ensure that the configs are reloaded successfully.

Re: Defining fields in transforms.conf

kmattern — Wed, 09 Nov 2011 17:19:21 GMT

Yep! Been there and done that. I always remember to stop and restart Splunk.

Re: Defining fields in transforms.conf

tgow — Wed, 09 Nov 2011 17:53:44 GMT

You will have to backslash out the "^" because it is an anchor in REGEX.

I took your log snippet and used the following:

props.conf

[source::...timefile...]
sourcetype = timefile
REPORT-time = time

transforms.conf

[time]
DELIMS = "\^"
FIELDS = DateAssign,TimeAssign,DateResolv,TimeResolv,CallID,Assignee,Resolution,TotalAsgnmntTime

and now it is working fine creating the fields.

If this helps don't forget to give me some points.

Re: Defining fields in transforms.conf

Jon_Webster — Wed, 09 Nov 2011 18:42:09 GMT

Here's a quick tip. You don't have to restart Splunk for changes to the .conf files, you can use the extract command with the reload=true option as documented in the Search command cheat sheet: http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/SearchCheatsheet

Extract field/value pairs and reload field extraction settings from disk.
... | extract reload=true

Re: Defining fields in transforms.conf

Drainy — Thu, 10 Nov 2011 09:05:45 GMT

Good shout!

Re: Defining fields in transforms.conf

Drainy — Thu, 10 Nov 2011 09:06:56 GMT

This isn't generally recommended. Although documented it doesn't actually always work very effectively. I have wasted many an hour thinking my configs were at fault when in fact they were fine but Splunk wasn't reloading them! 😉 Best practice would be to restart the system to be sure