Defining fields in transforms.conf

[kmattern]

Splunk Version 4.0.11

I’m working on getting Splunk to consume “log” files that I have dumped from a SQL Server helpdesk database. There are a lot of date and time fields in the events as well as dates and times that are included in the free-form text fields. I have managed to extract fields for my journal logs. My assignment log looks like this

2011-10-03^09:15:37^ ^ ^00021361^Brandon^ ^ ^

2011-10-03^08:10:35^2011-10-03^14:51:43^00021584^Bonnie^Completed^15^

2011-10-03^08:25:28^2011-10-04^07:17:13^00021585^Bonnie^Completed^20^

2011-10-03^08:40:19^2011-10-05^13:40:38^00021587^Bonnie^Completed^30^

2011-10-03^08:59:21^2011-10-04^07:15:03^00021588^Bonnie^Completed^20^

2011-10-03^09:14:46^2011-10-03^14:05:20^00021589^Patrick^Completed^15^

2011-10-03^09:24:33^2011-10-04^07:02:54^00021591^Stephanie^Completed^10^

2011-10-03^10:21:26^2011-10-03^14:25:04^00021592^Jeff^Completed^15^

2011-10-03^10:42:09^2011-10-03^10:42:25^00021593^Robert^Completed^10^

transforms.conf looks like this

[AsgLog] 

DELIMS="^" 

FIELDS="DateAssign",TimeAssign","DateResolv","TimeResolv","CallID","Assignee","Resolution","TotalAsgnmntTime" 

Splunk consumes the data and I can search it but the fields defined in transforms.conf are not available. I’ve tried extracting the fields but with two date fields and two time fields Splunk doesn’t seem able to discriminate between them.
What am I doing wrong?

[Jon_Webster]

Here's a quick tip. You don't have to restart Splunk for changes to the .conf files, you can use the extract command with the reload=true option as documented in the Search command cheat sheet: http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/SearchCheatsheet

Extract field/value pairs and reload field extraction settings from disk.
... | extract reload=true

[Drainy]

This isn't generally recommended. Although documented it doesn't actually always work very effectively. I have wasted many an hour thinking my configs were at fault when in fact they were fine but Splunk wasn't reloading them! 😉 Best practice would be to restart the system to be sure

[tgow]

You will have to backslash out the "^" because it is an anchor in REGEX.

I took your log snippet and used the following:

props.conf

[source::...timefile...]
sourcetype = timefile
REPORT-time = time

transforms.conf

[time]
DELIMS = "\^"
FIELDS = DateAssign,TimeAssign,DateResolv,TimeResolv,CallID,Assignee,Resolution,TotalAsgnmntTime

and now it is working fine creating the fields.

If this helps don't forget to give me some points.

[Drainy]

Good shout!

[Drainy]

Have you tried it with the FIELDS not surrounded by " "'s. E.g.

FIELDS=DateAssign,TimeAssign,DateResolv,TimeResolv,CallID,Assignee,Resolution,TotalAsgnmntTime 

Also, have you restarted Splunk? this is required to ensure that the configs are reloaded successfully.

[kmattern]

Yep! Been there and done that. I always remember to stop and restart Splunk.