Getting Data In

effective way to parse similar pattern

Path Finder

I have log which is printing value of an API in this manner
getCall=144:144:1:144:144 where I am parsing the value like this and getting desire values for this call

getCall=(?\d+):(?\d+):(?\d+):(?\d+):(?\d+)

I want to know if there is effective way to calculate this values, as I have ton's of api, adding this logic and building is not a great solution. please share your thought to optimize this query so that it will be applicable to all calls

Tags (2)
0 Karma

Path Finder

Please check my last comment, it is just picking first value, not all.

0 Karma

Path Finder

I found intersting observation not sure, if that is way how splunk works

[apifieldsmin]
WRITE
META = true
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1min::$5
MV
ADD = 1
Not able to parse MV_ADD option, and showing only first occurrence only.

[apifieldsmin1]
WRITE
META = true
REGEX = (\w+):(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1min::$5
MV
ADD = 1
Not able to parse MV_ADD option, and showing only first occurrence only.

Not sure for key=value pair I have to add some special multi param in transform file, so that same pattern can be allowed.

-Sumit

0 Karma

SplunkTrust
SplunkTrust

MV_ADD is only valid for search-time extractions, see http://docs.splunk.com/Documentation/Splunk/6.2.1/Admin/transformsconf for reference.

0 Karma

Path Finder

hhmmm

so this solution doesn't work at all, as we are doing this at index time 😞
any other approach, or should i raise to to splunk support team.

-Sumit

0 Karma

Path Finder

My Bad, I thought indexer has been restarted. but it was not, I have restarted and things are started working as we thought. on slide node, underscore or ., both are working.

Sorry about confusion and thanks a lot.

-Sumit

0 Karma

Path Finder

Yes, I did. I am giving configuration too, which I have set.

Transform.conf

[apifieldstotal]
WRITE
META = true
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1.total::$2

[apifieldsavg]
WRITE
META = true
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1_avg::$3

[apifieldscount]
WRITE
META = true
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1.count::$4

[apifieldsmin]
WRITE
META = true
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1_min::$5

[apifieldsmax]
WRITE
META = true
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1_max::$6

props.conf
[fdpacq]
pulldowntype = 1
TRANSFORMS-apifields = apifields
total,apifieldsavg,apifieldscount,apifieldsmin,apifieldsmax

0 Karma

SplunkTrust
SplunkTrust

Have your logs produce data in key=value format the way you'll be using it later, then Splunk will do all the extractions for you out of the box.

Alternatively, you could produce JSON data that describes itself.

For extracting those fields at index-time(!!), use this:

props.conf:

[your_sourcetype]
...
TRANSFORMS-apifields = apifields_total,apifields_avg,apifields_count,apifields_min,apifields_max

transforms.conf

[apifields_total]
WRITE_META = true
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1_total::$2

[apifields_avg]
WRITE_META = true
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1_avg::$3

[apifields_count]
WRITE_META = true
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1_count::$4

[apifields_min]
WRITE_META = true
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1_min::$5

[apifields_max]
WRITE_META = true
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1_max::$6
0 Karma

SplunkTrust
SplunkTrust

Did you restart the indexers and look at data indexed after that?

0 Karma

SplunkTrust
SplunkTrust

I see the issue now, concatenated FORMAT values only work for index-time extractions. I have updated my configuration post for that. Note, this will only take effect for data indexed after setting this.

0 Karma

Path Finder

We have made change on indexer, and still not able to see any result. sorry to coming late on this, but I don't have access to make this change and has to work with splunk admin team.

0 Karma

Path Finder

transform
[apifieldstotaltime]
CLEAN
KEYS = 1
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1.totaltime::$2
SOURCEKEY = _raw
MV
ADD = 1

[apifieldsavgtime]
CLEAN
KEYS = 1
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1avgtime::$3
SOURCE
KEY = getDataStack
MV_ADD = 1

[apifieldstotalno]
CLEAN
KEYS = 1
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1totalno::$4
SOURCE
KEY = getDataStack
MV_ADD = 1

[apifieldsmintime]
CLEAN
KEYS = 1
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1mintime::$5
SOURCE
KEY = getDataStack
MV_ADD = 1

[apifieldsmaxtime]
CLEAN
KEYS = 1
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1maxtime::$6
SOURCE
KEY = getDataStack
MV_ADD = 1

props
[fdpacq]
REPORT-REPORT-getDataAcq = extractserverStack,apifieldstotaltime,apifieldsavgtime,apifieldstotalno,apifieldsmintime,apifields_maxtime

0 Karma

Path Finder

No success ..
I have same configuration, I have multiple of this values in and event
I feel somehow, adding to extracted value with string causing issue ..

0 Karma

SplunkTrust
SplunkTrust

The underscore might be confusing the $1, try $1.total etc.

0 Karma

Path Finder

I have done above formatting but it is not working as we are expecting. though it is putting value on total,avg,min and max but not with $1. Not sure formatting is concat $1 and string value ??
BTW, my other pattern are working fine where I am assigning the value as $1::$2

0 Karma

SplunkTrust
SplunkTrust

I see, you're trying to extract the first part of the field name from the event and infer the second part of the field name from the position within the event? Try this:

props.conf:

[your_sourcetype]
...
TRANSFORMS-apifields = apifields_total,apifields_avg,apifields_count,apifields_min,apifields_max

transforms.conf

[apifields_total]
WRITE_META = true
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1_total::$2

[apifields_avg]
WRITE_META = true
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1_avg::$3

[apifields_count]
WRITE_META = true
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1_count::$4

[apifields_min]
WRITE_META = true
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1_min::$5

[apifields_max]
WRITE_META = true
REGEX = (\w+)=(\d+):(\d+):(\d+):(\d+):(\d+)
FORMAT = $1_max::$6
0 Karma

Path Finder

This time is it not about key and value, We are doing this, but for any API getting this value using key-pair will take long log message. hence we made little effective and producing this in above manner.
I am doing below parsing

getOAuthToken=(?(getOauthtotalExecTime>\d+):(?(getOauthavgExecTime>\d+):(?(getOauthnoOfExecutions>\d+):(?(getOauthminTime>\d+):(?(getOauth_maxTime>\d+)

I want a way to effectively use the api name and associate with new key values like.

getCallTotalTime
getCall
AvgTime
getCallAvgTime
getCall
Occurance
getCallMInTime
getCall
MaxTime

0 Karma