topic Re: How to define sourcetype for new file format in Splunk Search

How to define sourcetype for new file format

bowesmana — Tue, 10 May 2011 06:37:14 GMT

I'm a newbie, so go easy please.

I have a file structure that has field groups separated by ; and within each group there is 1 or more values. For example the line

000101;102;356;3284;4,2;0;UserName,UserId;2011-05-02 07:24:25.275;63,15;0,9;id=xx,q=name

equates to the following

Service = 000101

Cmd = 102

Pid = 356

Tid = 3284

Counts = 4,2 (2 different counters)

Result = 0

User = UserName,Id

Date = 2011-05-02 07:24:25.275

--- All data records are constant syntax up to this point, followed by

Timers = 63,15 - contains 1..n timers - first is total elapsed time

Values = 0,9 - contains 1..n values related to Cmd 102

Fields = id=xx,q=name (contains 1..n key/value pairs - related to Cmd 102)

I've read lots about sourcetypes, props.conf, inputs.conf, index time and search time and so on but just can't quite join the dots to work out how to make splunk extract the data and assign it to these named fields.

Can anyone start me with the first dot and point me in the right direction

Thanks

Re: How to define sourcetype for new file format

gkanapathy — Tue, 10 May 2011 14:02:23 GMT

In props.conf (just put it on all machines, forwarders, indexers, and search heads; look here if you really want the details) put:

props.conf:

[mynewsourcetype]
SHOULD_LINEMERGE = false
TIME_PREFIX=^(?:[^;]*;){7}
TIME_FORMAT=%Y-%m-%d %H:%M:%S.%3N
REPORT-mynewsourcetypefields = mynewsourcetypefields
KV_MODE = auto

transforms.conf could be one of multiple possible versions. Simple one is:

[mynewsourcetypefields]
DELIMS=";,"
FIELDS = Service,Cmd,Pid,Tid,Count1,Count2,UserName,UserId,DateTime

A more complete one is:

[mynewsourcetypefields]
REGEX = ^(?<Service>[^;]*);(?<Cmd>[^;]*);(?<Pid>[^;]*);(?<Tid>[^;]*)(?<Count1>[^,]*),(?<Count2>[^;]*);(?<UserName>[^,]*),(?<UserId>[^;]*);(?<DateTime>[^;]*);(?:(?<Timers>[^;]*);)?(?:(?<Values>[^;]*);)?

Re: How to define sourcetype for new file format

bowesmana — Wed, 11 May 2011 02:18:43 GMT

Update....

After a bit of mistakes and fiddling with props.conf and transforms.conf in etc/system/local, I got the fields to be recognised, by setting a manual sourcetype=server_instr when adding data. However, I also need to be able to split the Timers, Values and Fields depending on certain criteria in the data, e.g. if Cmd=102, the if Value[0] == 3 then count of Timers = X, so how can I do this. I also want to search, e.g. Cmd=102 && Timer[4] > 3000. How can I do this?

Re: How to define sourcetype for new file format

bowesmana — Wed, 11 May 2011 02:37:36 GMT

Update....
In transforms.conf, for Date onwards, I have

....;(?[^;]);(?:(?[^;]);)?(?:(?[^;]);)?(?:(?[^;]);)

I don't fully understand the regex, but I am never getting any 'Values' field found and in a line that has

1672,32,0,1640;0,0,1,1,42;

which represents 1672,32,0,1640 as Timers and 0,0,1,1,42 as Values and there are no Fields, Splunk says that 1672,32,0,1640 are Timers and 0,0,1,1,42 are Fields, but no Values, so can anyone say what's wrong with the regex above? In my original post I said that Values/Fields can be 1..n, but it's 0..n

Re: How to define sourcetype for new file format

bowesmana — Wed, 11 May 2011 06:54:25 GMT

Update....

I finally got the regex going

REGEX = ^(?[^;]);(?[^;]);(?[^;]);(?[^;]);(?[^,]),(?[^;]);(?[^;]);(?[^,]),(?[^;]);(?[^;]);(?:(?[^;]);)?(?:(?[^;]);)?(?:(?[^;]*)$)?

but is there a way to use the Timers and Values fields as an array of values within those fields, so they can be referenced in such a way as Timers[3] or will I have to use rex during the search to make dynamic fields from the Timers field?

Re: How to define sourcetype for new file format

gkanapathy — Wed, 11 May 2011 13:36:34 GMT

you can use the mvindex() eval function on your multi-value lists, but you first need to convert it from a string to a mvlist, using either the makemv search command, the split() eval function, or by defining an additional REPORT extraction (that runs after the main one you already have) for each field that operates on the originallly extracted list and puts the values in a list with the MV_ADD option.

Re: How to define sourcetype for new file format

bowesmana — Thu, 12 May 2011 03:29:33 GMT

Thanks gkanapathy, great help - so many ways to achieve the same thing, wow this is powerful.