Getting Data In

Defining fields in transforms.conf

Builder

Splunk Version 4.0.11

I’m working on getting Splunk to consume “log” files that I have dumped from a SQL Server helpdesk database. There are a lot of date and time fields in the events as well as dates and times that are included in the free-form text fields. I have managed to extract fields for my journal logs. My assignment log looks like this


2011-10-03^09:15:37^ ^ ^00021361^Brandon^ ^ ^
2011-10-03^08:10:35^2011-10-03^14:51:43^00021584^Bonnie^Completed^15^
2011-10-03^08:25:28^2011-10-04^07:17:13^00021585^Bonnie^Completed^20^
2011-10-03^08:40:19^2011-10-05^13:40:38^00021587^Bonnie^Completed^30^
2011-10-03^08:59:21^2011-10-04^07:15:03^00021588^Bonnie^Completed^20^
2011-10-03^09:14:46^2011-10-03^14:05:20^00021589^Patrick^Completed^15^
2011-10-03^09:24:33^2011-10-04^07:02:54^00021591^Stephanie^Completed^10^
2011-10-03^10:21:26^2011-10-03^14:25:04^00021592^Jeff^Completed^15^
2011-10-03^10:42:09^2011-10-03^10:42:25^00021593^Robert^Completed^10^

transforms.conf looks like this


[AsgLog]
DELIMS="^"
FIELDS="DateAssign",TimeAssign","DateResolv","TimeResolv","CallID","Assignee","Resolution","TotalAsgnmntTime"

Splunk consumes the data and I can search it but the fields defined in transforms.conf are not available. I’ve tried extracting the fields but with two date fields and two time fields Splunk doesn’t seem able to discriminate between them.
What am I doing wrong?

Tags (1)
0 Karma

Splunk Employee
Splunk Employee

Here's a quick tip. You don't have to restart Splunk for changes to the .conf files, you can use the extract command with the reload=true option as documented in the Search command cheat sheet: http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/SearchCheatsheet

Extract field/value pairs and reload field extraction settings from disk.
... | extract reload=true

0 Karma

Champion

This isn't generally recommended. Although documented it doesn't actually always work very effectively. I have wasted many an hour thinking my configs were at fault when in fact they were fine but Splunk wasn't reloading them! 😉 Best practice would be to restart the system to be sure

0 Karma

Splunk Employee
Splunk Employee

You will have to backslash out the "^" because it is an anchor in REGEX.

I took your log snippet and used the following:

props.conf

[source::...timefile...]
sourcetype = timefile
REPORT-time = time

transforms.conf

[time]
DELIMS = "\^"
FIELDS = DateAssign,TimeAssign,DateResolv,TimeResolv,CallID,Assignee,Resolution,TotalAsgnmntTime

and now it is working fine creating the fields.

If this helps don't forget to give me some points.

Champion

Good shout!

0 Karma

Champion

Have you tried it with the FIELDS not surrounded by " "'s. E.g.

FIELDS=DateAssign,TimeAssign,DateResolv,TimeResolv,CallID,Assignee,Resolution,TotalAsgnmntTime 

Also, have you restarted Splunk? this is required to ensure that the configs are reloaded successfully.

0 Karma

Builder

Yep! Been there and done that. I always remember to stop and restart Splunk.

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!