Splunk Search

how to extract these fields using regular expression

prashanthberam
Explorer

"cvpEditAction" : "R",
"cvpEditAllowedAmount" : 333.57,

Could someone please help me how to extract these?
thanks.

0 Karma

aaraneta_splunk
Splunk Employee
Splunk Employee

@prashanthberam - Looks like you have a few possible solutions to your question. If one of them provided a working solution, please don't forget to click "Accept" below the best answer to resolve this post. If you still need help, please leave a comment. Don’t forget to upvote anything that was helpful too. Thanks!

0 Karma

adayton20
Contributor

For the extractions you wanted from your original post, these should work:

| rex field=_raw "\"cvpEditAction\"\s\:\s\"(?P<cvpEditAction>.[^\"]*)"
| rex field=_raw "\"cvpEditAllowedAmount\"\s\:\s(?P<cvpEditAllowedAmount>.[^\,]*)"

As far as helping you understand how to build these, and for future instances:

I'd do some reading on regular expressions and how to use them in Splunk. Learning regular expressions is actually easier than it looks. I started with this site: https://regexone.com/. Then, I browsed some examples here on Splunk Answers and practiced on my own. Might help you too.

Looking over the log sample you provided, it appears you can use similar rex patterns for the other fields you want. It looks as if each field name (the key) has a double quote in the begging and end, and the field value (the key value) either has a double quote or a comma at the end of it. With that in mind, you can use the patterns I provided to interchange and extract other fields out of similar logs. There are several other ways to extract the fields, this is just the way I chose to do it because it is the way I learned. I'm not a regex master... yet.

Take note of the end part for each regex I provided to you.

| rex field=_raw "\"cvpEditAction\"\s\:\s\"(?P<cvpEditAction>.[^\"]*)"

.[^\"]*) basically says, extract everything up to a double quote.

and

| rex field=_raw "\"cvpEditAllowedAmount\"\s\:\s(?P<cvpEditAllowedAmount>.[^\,]*)"

.[^\,]*) basically says extract everything up to the comma.

For example, let’s say you want to extract the value of "cvpEditAddedLineCount" in your log sample.

Notice the value of "cvpEditAddedLineCount" is 0,. A comma appears at the end of your value. So you want to extract everything AFTER "cvpEditAddedLineCount" : but BEFORE the comma.

Modify the regex I provided for excluding the comma, like this:

| rex field=_raw "\"cvpEditAddedLineCount\"\s\:\s(?P<cvpEditAddedLineCount>.[^\,]*)"

Notice the only thing I changed was the text for the field name (the key). From cvpEditAllowedAmount to cvpEditAddedLineCount. Everything else stayed the same.

From

| rex field=_raw "\"cvpEditAllowedAmount\"\s\:\s(?P<cvpEditAllowedAmount>.[^\,]*)"

to

| rex field=_raw "\"cvpEditAddedLineCount\"\s\:\s(?P<cvpEditAddedLineCount>.[^\,]*)"

Likewise,

Say you want to extract the value of "cvpEditClaimNumber" from your sample log. Notice the value, "OK1-0201705480170030C-00" has double quotes wrapped around it.

Modify the regex with the quote exclusion, from:

| rex field=_raw "\"cvpEditClaimNumber\"\s\:\s\"(?P<cvpEditClaimNumber>.[^\"]*)"

to

| rex field=_raw "\"cvpEditClaimNumber\"\s\:\s\"(?P<cvpEditClaimNumber>.[^\"]*)"

If I’m not making any sense, let me know so I can clarify further.

Now, I’m assuming at some point you might want to extract the other fields from that log. Cluttering up your search with 30+ rex commands might get a bit cumbersome.

There are a few ways to have Splunk extract fields without specifying them directly in the search.

1.) Via the GUI under Settings > Fields > Field Extractions
2.) Via props.conf or props.conf AND transforms.conf

Via the GUI under Settings > Fields > Field Extractions, I used one of your fields as an example
alt text

If you intend on sharing this field extraction with other sourcetypes or with other people, you’ll need to edit the permissions of the field extraction after you save it.

Another way you can have Splunk extract fields for you is by directly editing props.conf of your app on your search head. In this case, I'm using the search app, which, for me is located at /opt/splunk/etc/apps/search/local. To extract the "cvpEditAction", you can add a stanza in the props.conf file that looks something like this:

[yourSourcetype]
EXTRACT-cvpEditAction = \"cvpEditAction\"\s\:\s\"(?P<cvpEditAction>.[^\"]*)

You can also use a combination of props.conf and transforms.conf, but for simple field extractions like this, I prefer keeping them consolidated under props.conf.

You don't necessary have to restart your search head after adding a field extraction either (although you can if you want). Simply type |extract reload=true into your search bar.

You can read more about extracting fields here:

http://docs.splunk.com/Documentation/Splunk/6.5.2/Knowledge/Aboutfields
http://docs.splunk.com/Documentation/Splunk/latest/Search/Extractfieldswithsearchcommands
https://docs.splunk.com/Documentation/Splunk/6.5.2/Scenarios/Extractfields

and about props.conf and transforms.conf here:

https://docs.splunk.com/Documentation/Splunk/6.5.2/Admin/Propsconf
http://docs.splunk.com/Documentation/Splunk/latest/admin/transformsconf

and of course, practice makes perfect!
https://regex101.com/
http://regexr.com/

Hope that helps.

woodcock
Esteemed Legend

Like this:

... | rex "(?ms)\"cvpEditAction\"\s*:\s*\"(?<cvpEditAction>[^\"]*\",[\r\n\s]*\"cvpEditAllowedAmount\"\s*:\s*(?<cvpEditAllowedAmount>[^,]*"

For automatic extractions, read about KV_MODE and KVPs here:
https://docs.splunk.com/Documentation/Splunk/6.5.2/Admin/Propsconf

0 Karma

adayton20
Contributor

Can you post the original raw log and your expected output after extraction?

0 Karma

prashanthberam
Explorer

2017-02-28 16:50:20.123 [WMQJCAResourceAdapter : 0] [DEBUG] [DCN 0201705480170030C] LoggerUtil - {
"cvpEditAction" : "R",
"cvpEditAddedLineCount" : 0,
"cvpEditAllowedAmount" : 333.57,
"cvpEditClaimNumber" : "OK1-0201705480170030C-00",
"cvpEditClaimSequenceNumber" : 3697289410194,
"cvpEditCreatedLineIndicator" : "F",
"cvpEditDefenseDescriptorID" : 516,
"cvpEditDefenseID" : 306,
"cvpEditFiringContextNumber" : 1067,
"cvpEditFiringOrdinal" : 1,
"cvpEditFiringStateNumber" : 0,
"cvpEditIntegrationScenario" : 2,
"cvpEditLineNumber" : "2",
"cvpEditModifierCode" : null,
"cvpEditMultipleProviderIndicator" : "F",
"cvpEditMultiplier" : null,
"cvpEditOrdinal" : 2,
"cvpEditParameterType" : "O",
"cvpEditPartitionCode" : "OK1",
"cvpEditPrimaryDefenseAuthor" : "Center for Medicare Services",
"cvpEditPrimaryDefenseLocation" : "National Correct Coding Policy Manual",
"cvpEditPrimaryDefenseText" : null,
"cvpEditProcedureCode" : "95940",
"cvpEditRank" : null,
"cvpEditRevenueCode" : null,
"cvpEditRuleCategoryID" : "0101000000",
"cvpEditRuleCategoryLevel1" : "Add On",
"cvpEditRuleCategoryLevel2" : "CPT",
"cvpEditRuleCategoryLevel3" : null,
"cvpEditRuleCategoryLevel4" : null,
"cvpEditRuleCategoryLevel5" : null,
"cvpEditSourceSegment" : "110",
"cvpEditText" : "THE PRIMARY SERVICE CODE IS ABSENT BECAUSE THE CODE WAS NOT SUBMITTED OR THE CODE WAS DENIED. PER CPT, ADD-ON SERVICES ARE PERFORMED ONLY AS AN ADDITION TO THE PRIMARY SERVICE OR PROCEDURE AND MUST NEVER BE REPORTED WITHOUT THE PRIMARY SERVICE CODE. THEREFORE, 95940 IS NOT REIMBURSABLE.",
"cvpEditTier" : null,
"cvpEditTriggerEventReference" : "OK1-0201705480170030C-00",
"cvpEditTriggerEventSequence" : 3697289410194,
"cvpEditUnitCount" : 8,
"cvpEditWorkQueue" : null,
"cvpEditDetailUtilization" : {
"cvpUtilizationAmount" : 0,

0 Karma

prashanthberam
Explorer

I need to extract cvpEditAction= R like this..please help me

0 Karma

illescas
Engager

Do these fields both exist in one event you have in splunk? If so here is my quick (definitely not the best) regex to extract the fields values

... | rex field=_raw ".*cvpEditAction.*\s:\s\"(?<cvpEditAction>.*)\".*\"cvpEditAllowedAmount.*\s:\s(?<cvpEditAllowedAmount>.*).*"

And in your Splunk search Interesting Fields you should see cvpEditAction and cvpEditAllowedAmount

0 Karma
Get Updates on the Splunk Community!

Cloud Platform & Enterprise: Classic Dashboard Export Feature Deprecation

As of Splunk Cloud Platform 9.3.2408 and Splunk Enterprise 9.4, classic dashboard export features are now ...

Explore the Latest Educational Offerings from Splunk (November Releases)

At Splunk Education, we are committed to providing a robust learning experience for all users, regardless of ...

New This Month in Splunk Observability Cloud - Metrics Usage Analytics, Enhanced K8s ...

The latest enhancements across the Splunk Observability portfolio deliver greater flexibility, better data and ...