topic Re: effective way to parse similar pattern in Getting Data In

effective way to parse similar pattern

sumitnagal — Sat, 08 Nov 2014 09:37:06 GMT

I have log which is printing value of an API in this manner
getCall=144:144:1:144:144 where I am parsing the value like this and getting desire values for this call

getCall=(?\d+):(?\d+):(?\d+):(?\d+):(?\d+)

I want to know if there is effective way to calculate this values, as I have ton's of api, adding this logic and building is not a great solution. please share your thought to optimize this query so that it will be applicable to all calls

Re: effective way to parse similar pattern

martin_mueller — Sat, 08 Nov 2014 10:54:18 GMT

Have your logs produce data in key=value format the way you'll be using it later, then Splunk will do all the extractions for you out of the box.

Alternatively, you could produce JSON data that describes itself.

For extracting those fields at index-time(!!), use this:

props.conf:

[your_sourcetype]
...
TRANSFORMS-apifields = apifields_total,apifields_avg,apifields_count,apifields_min,apifields_max

transforms.conf

[apifields_total]
WRITE_META = true
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1_total::$2

[apifields_avg]
WRITE_META = true
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1_avg::$3

[apifields_count]
WRITE_META = true
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1_count::$4

[apifields_min]
WRITE_META = true
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1_min::$5

[apifields_max]
WRITE_META = true
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1_max::$6

Re: effective way to parse similar pattern

sumitnagal — Mon, 28 Sep 2020 18:06:57 GMT

This time is it not about key and value, We are doing this, but for any API getting this value using key-pair will take long log message. hence we made little effective and producing this in above manner.
I am doing below parsing

getOAuthToken=(?(getOauth_totalExecTime>\d+):(?(getOauth_avgExecTime>\d+):(?(getOauth_noOfExecutions>\d+):(?(getOauth_minTime>\d+):(?(getOauth_maxTime>\d+)

I want a way to effectively use the api name and associate with new key values like.

getCall_TotalTime
getCall_AvgTime
getCall_AvgTime
getCall_Occurance
getCall_MInTime
getCall_MaxTime

Re: effective way to parse similar pattern

martin_mueller — Sat, 08 Nov 2014 20:14:00 GMT

I see, you're trying to extract the first part of the field name from the event and infer the second part of the field name from the position within the event? Try this:

props.conf:

[your_sourcetype]
...
TRANSFORMS-apifields = apifields_total,apifields_avg,apifields_count,apifields_min,apifields_max

transforms.conf

[apifields_total]
WRITE_META = true
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1_total::$2

[apifields_avg]
WRITE_META = true
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1_avg::$3

[apifields_count]
WRITE_META = true
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1_count::$4

[apifields_min]
WRITE_META = true
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1_min::$5

[apifields_max]
WRITE_META = true
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1_max::$6

Re: effective way to parse similar pattern

sumitnagal — Sat, 08 Nov 2014 21:26:01 GMT

I have done above formatting but it is not working as we are expecting. though it is putting value on total,avg,min and max but not with $1. Not sure formatting is concat $1 and string value ??
BTW, my other pattern are working fine where I am assigning the value as $1::$2

Re: effective way to parse similar pattern

martin_mueller — Sat, 08 Nov 2014 22:17:53 GMT

The underscore might be confusing the $1, try $1.total etc.

Re: effective way to parse similar pattern

sumitnagal — Sat, 08 Nov 2014 23:43:09 GMT

No success ..
I have same configuration, I have multiple of this values in and event
I feel somehow, adding to extracted value with string causing issue ..

Re: effective way to parse similar pattern

sumitnagal — Mon, 28 Sep 2020 18:07:00 GMT

transform
[apifields_totaltime]
CLEAN_KEYS = 1
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1.totaltime::$2
SOURCE_KEY = _raw
MV_ADD = 1

[apifields_avgtime]
CLEAN_KEYS = 1
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1_avgtime::$3
SOURCE_KEY = getDataStack
MV_ADD = 1

[apifields_totalno]
CLEAN_KEYS = 1
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1_totalno::$4
SOURCE_KEY = getDataStack
MV_ADD = 1

[apifields_mintime]
CLEAN_KEYS = 1
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1_mintime::$5
SOURCE_KEY = getDataStack
MV_ADD = 1

[apifields_maxtime]
CLEAN_KEYS = 1
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1_maxtime::$6
SOURCE_KEY = getDataStack
MV_ADD = 1

props
[fdpacq]
REPORT-REPORT-getDataAcq = extractserverStack,apifields_totaltime,apifields_avgtime,apifields_totalno,apifields_mintime,apifields_maxtime

Re: effective way to parse similar pattern

martin_mueller — Sun, 09 Nov 2014 00:19:31 GMT

I see the issue now, concatenated FORMAT values only work for index-time extractions. I have updated my configuration post for that. Note, this will only take effect for data indexed after setting this.

Re: effective way to parse similar pattern

sumitnagal — Fri, 12 Dec 2014 07:08:10 GMT

We have made change on indexer, and still not able to see any result. sorry to coming late on this, but I don't have access to make this change and has to work with splunk admin team.

Re: effective way to parse similar pattern

martin_mueller — Fri, 12 Dec 2014 10:26:39 GMT

Did you restart the indexers and look at data indexed after that?

Re: effective way to parse similar pattern

sumitnagal — Mon, 28 Sep 2020 18:27:53 GMT

Yes, I did. I am giving configuration too, which I have set.

Transform.conf

[apifields_total]
WRITE_META = true
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1.total::$2

[apifields_avg]
WRITE_META = true
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1_avg::$3

[apifields_count]
WRITE_META = true
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1.count::$4

[apifields_min]
WRITE_META = true
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1_min::$5

[apifields_max]
WRITE_META = true
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1_max::$6

props.conf
[fdpacq]
pulldown_type = 1
TRANSFORMS-apifields = apifields_total,apifields_avg,apifields_count,apifields_min,apifields_max

Re: effective way to parse similar pattern

sumitnagal — Wed, 17 Dec 2014 07:11:50 GMT

My Bad, I thought indexer has been restarted. but it was not, I have restarted and things are started working as we thought. on slide node, underscore or ., both are working.

Sorry about confusion and thanks a lot.

-Sumit

Re: effective way to parse similar pattern

sumitnagal — Mon, 28 Sep 2020 18:28:23 GMT

I found intersting observation not sure, if that is way how splunk works

[apifields_min]
WRITE_META = true
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1_min::$5
MV_ADD = 1
Not able to parse MV_ADD option, and showing only first occurrence only.

[apifields_min1]
WRITE_META = true
REGEX = (\w+):(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1_min::$5
MV_ADD = 1
Not able to parse MV_ADD option, and showing only first occurrence only.

Not sure for key=value pair I have to add some special multi param in transform file, so that same pattern can be allowed.

-Sumit

Re: effective way to parse similar pattern

sumitnagal — Sat, 17 Jan 2015 00:30:16 GMT

Please check my last comment, it is just picking first value, not all.

Re: effective way to parse similar pattern

martin_mueller — Sat, 17 Jan 2015 00:32:46 GMT

MV_ADD is only valid for search-time extractions, see http://docs.splunk.com/Documentation/Splunk/6.2.1/Admin/transformsconf for reference.

Re: effective way to parse similar pattern

sumitnagal — Wed, 28 Jan 2015 04:41:51 GMT

hhmmm

so this solution doesn't work at all, as we are doing this at index time 😞
any other approach, or should i raise to to splunk support team.

-Sumit