effective way to parse similar pattern

[sumitnagal]

I have log which is printing value of an API in this manner
getCall=144:144:1:144:144 where I am parsing the value like this and getting desire values for this call

getCall=(?\d+):(?\d+):(?\d+):(?\d+):(?\d+)

I want to know if there is effective way to calculate this values, as I have ton's of api, adding this logic and building is not a great solution. please share your thought to optimize this query so that it will be applicable to all calls

[sumitnagal]

Please check my last comment, it is just picking first value, not all.

[sumitnagal]

I found intersting observation not sure, if that is way how splunk works

[apifields_min]
WRITE_META = true
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1_min::$5
MV_ADD = 1
Not able to parse MV_ADD option, and showing only first occurrence only.

[apifields_min1]
WRITE_META = true
REGEX = (\w+):(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1_min::$5
MV_ADD = 1
Not able to parse MV_ADD option, and showing only first occurrence only.

Not sure for key=value pair I have to add some special multi param in transform file, so that same pattern can be allowed.

-Sumit

[martin_mueller]

MV_ADD is only valid for search-time extractions, see http://docs.splunk.com/Documentation/Splunk/6.2.1/Admin/transformsconf for reference.

[sumitnagal]

hhmmm

so this solution doesn't work at all, as we are doing this at index time 😞
any other approach, or should i raise to to splunk support team.

-Sumit

[sumitnagal]

My Bad, I thought indexer has been restarted. but it was not, I have restarted and things are started working as we thought. on slide node, underscore or ., both are working.

Sorry about confusion and thanks a lot.

-Sumit

[sumitnagal]

Yes, I did. I am giving configuration too, which I have set.

Transform.conf

[apifields_total]
WRITE_META = true
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1.total::$2

[apifields_avg]
WRITE_META = true
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1_avg::$3

[apifields_count]
WRITE_META = true
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1.count::$4

[apifields_min]
WRITE_META = true
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1_min::$5

[apifields_max]
WRITE_META = true
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1_max::$6

props.conf
[fdpacq]
pulldown_type = 1
TRANSFORMS-apifields = apifields_total,apifields_avg,apifields_count,apifields_min,apifields_max

[martin_mueller]

Have your logs produce data in key=value format the way you'll be using it later, then Splunk will do all the extractions for you out of the box.

Alternatively, you could produce JSON data that describes itself.

For extracting those fields at index-time(!!), use this:

props.conf:

[your_sourcetype]
...
TRANSFORMS-apifields = apifields_total,apifields_avg,apifields_count,apifields_min,apifields_max

transforms.conf

[apifields_total]
WRITE_META = true
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1_total::$2

[apifields_avg]
WRITE_META = true
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1_avg::$3

[apifields_count]
WRITE_META = true
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1_count::$4

[apifields_min]
WRITE_META = true
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1_min::$5

[apifields_max]
WRITE_META = true
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1_max::$6

[martin_mueller]

Did you restart the indexers and look at data indexed after that?

[martin_mueller]

I see the issue now, concatenated FORMAT values only work for index-time extractions. I have updated my configuration post for that. Note, this will only take effect for data indexed after setting this.

[sumitnagal]

We have made change on indexer, and still not able to see any result. sorry to coming late on this, but I don't have access to make this change and has to work with splunk admin team.

[sumitnagal]

transform
[apifields_totaltime]
CLEAN_KEYS = 1
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1.totaltime::$2
SOURCE_KEY = _raw
MV_ADD = 1

[apifields_avgtime]
CLEAN_KEYS = 1
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1_avgtime::$3
SOURCE_KEY = getDataStack
MV_ADD = 1

[apifields_totalno]
CLEAN_KEYS = 1
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1_totalno::$4
SOURCE_KEY = getDataStack
MV_ADD = 1

[apifields_mintime]
CLEAN_KEYS = 1
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1_mintime::$5
SOURCE_KEY = getDataStack
MV_ADD = 1

[apifields_maxtime]
CLEAN_KEYS = 1
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1_maxtime::$6
SOURCE_KEY = getDataStack
MV_ADD = 1

props
[fdpacq]
REPORT-REPORT-getDataAcq = extractserverStack,apifields_totaltime,apifields_avgtime,apifields_totalno,apifields_mintime,apifields_maxtime

[sumitnagal]

No success ..
I have same configuration, I have multiple of this values in and event
I feel somehow, adding to extracted value with string causing issue ..

[martin_mueller]

The underscore might be confusing the $1, try $1.total etc.

[sumitnagal]

I have done above formatting but it is not working as we are expecting. though it is putting value on total,avg,min and max but not with $1. Not sure formatting is concat $1 and string value ??
BTW, my other pattern are working fine where I am assigning the value as $1::$2

[martin_mueller]

I see, you're trying to extract the first part of the field name from the event and infer the second part of the field name from the position within the event? Try this:

props.conf:

[your_sourcetype]
...
TRANSFORMS-apifields = apifields_total,apifields_avg,apifields_count,apifields_min,apifields_max

transforms.conf

[apifields_total]
WRITE_META = true
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1_total::$2

[apifields_avg]
WRITE_META = true
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1_avg::$3

[apifields_count]
WRITE_META = true
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1_count::$4

[apifields_min]
WRITE_META = true
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1_min::$5

[apifields_max]
WRITE_META = true
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1_max::$6

[sumitnagal]

This time is it not about key and value, We are doing this, but for any API getting this value using key-pair will take long log message. hence we made little effective and producing this in above manner.
I am doing below parsing

getOAuthToken=(?(getOauth_totalExecTime>\d+):(?(getOauth_avgExecTime>\d+):(?(getOauth_noOfExecutions>\d+):(?(getOauth_minTime>\d+):(?(getOauth_maxTime>\d+)

I want a way to effectively use the api name and associate with new key values like.

getCall_TotalTime
getCall_AvgTime
getCall_AvgTime
getCall_Occurance
getCall_MInTime
getCall_MaxTime