topic Re: How to extract fields and assign values from my data with the field extractor utility? in Splunk Search

How to extract fields and assign values from my data with the field extractor utility?

gmelasecca — Fri, 24 Apr 2015 16:14:06 GMT

I have a custom file which we don't have problems searching certain "strings" within, but what I cannot figure out is how to create a custom field and then assign values to search on it. For instance, below is the query used to search for the 'string' with any IP Address thrown in between the single quotes at the end. I do not have issues returning the search. The issue is i want to use the field extractor to create fields. SEARCH: index=INDEX sourcetype=INDEX_LOG "call failed: Unable to connect to server '*'"

Field Example: create a field called: "Unable to connect to server" and within this field, i would be able to pull the IP Address values from it, distinguish the unique or duplicate values in a certain time frame. I hope that makes sense, i am new to splunk.

SAMPLE LOG FILE in bold is the search term i'm mainly looking for:
09104464 5160 AB9D87B12528D94D8CEFD068DA0C2B48 AEJ00101/PRD20002 REQUEST 0 BPSEJLA1/290 UpdImgSt
09104465 4108 9BC9B8192CFA92459E2353A4DABE24C6 AMN00101/PRD20002 REQUEST 0 BPSMONA1/349 RecAlertsWKS - WWS05656

09104465 4108 9BC9B8192CFA92459E2353A4DABE24C6 WSF05656/DEFAULT E G R PGNP0008 10060 BPSMONA1/2698 Error: Windows API function 'connect' call failed: Unable to connect to server '12.34.56.78'.,OT:100037/GMT-04,Suppressed=3

09104465 4108 9BC9B8192CFA92459E2353A4DABE24C6 WSF05656/DEFAULT C N R PGNS0001 10060 BPSMONA1/2698 Network Transport Error: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond...,OT:100037/GMT-04,Suppressed=3
09104467 3496 8758A6696193044980CF175B2A678315 ABC00101/PRD20002 REQUEST 0 BPSBCLA1/714 OprUpdSt
09104479 3372 93DA8C8A149B464590B928F80657C978 AOBCL101/PRD20002 SPCLTRCE 3 BPGCONTX/732 Data Insert Error
09104483 3496 94B1FAB4D411B943A9CE71B93EF7E752 ABC00101/PRD20002 REQUEST 0 BPSBCLA1/714 ProcReq
09

Re: How to extract fields and assign values from my data with the field extractor utility?

stephanefotso — Fri, 24 Apr 2015 16:35:54 GMT

Hello!
Try this

index=INDEX sourcetype=INDEX_LOG....... |rex field=_raw "^(?:[^'\n]*'){3}(?P<Unable to connect to server>[^']+)"|table "Unable to connect to server"

or this

index=INDEX sourcetype=INDEX_LOG....... |rex field=_raw "^(?:[^'\n]*'){3}(?P<Unable_to_connect_to_server>\d+\.\d+\.\d+\.\d+)"|table Unable_to_connect_to_server

Re: How to extract fields and assign values from my data with the field extractor utility?

rsennett_splunk — Fri, 24 Apr 2015 17:21:21 GMT

Let's walk through the events and then narrow down how to get to the values you want.
Grab your sample, and use regex101.com to see what I see.

grab your sample and then put this regex in and turn on the g flag for global:

\d{8}\s+\d{4}\s+[^\s]+\s[^\s]+

You can see that the events match up to that point, so to be smart about it, again, more sample data would tell us a lot.
Meanwhile you have something to grab on to:

Error:

So to grab this in a field you would use:

Error:\s(?P<error>.+)

Because you have the target to anchor on "Error:" it doesn't matter where it is in the event...

Then to grab the IP address you can use rex in line as follows rex field=error "(?Pd+\.\d+\.\d+\.\d+)"

Or you can put it in props.conf with this format:

EXTRACT-detail (?P<msg>unable.*) in error

if you update the question with events that have errors, I can advise you further... but basically you're looking for something to hang on to.

For efficiency... you would grab the ERROR field as a search time extraction and then start your searches with
index=blah sourcetype=foo error=*|... or index=blah sourcetype=foo|search error=*|...

Re: How to extract fields and assign values from my data with the field extractor utility?

gmelasecca — Fri, 24 Apr 2015 18:35:36 GMT

Unfortunately the first query resulted in a code error:
Error in 'rex' command: Encountered the following error while compiling the regex '^(?:[^'\n]*'){3}(?P[^']+)': Regex: syntax error in sub-pattern name (missing terminator)

the second query did not return the results expected. it actually just returned the first portion of my search:
"call failed: Unable to connect to server '*'" than displayed exactly the same as if i ran my original query.

any thoughts? i can take a screen shot or show what was returned if needed.

Thanks

Re: How to extract fields and assign values from my data with the field extractor utility?

rsennett_splunk — Fri, 24 Apr 2015 19:32:06 GMT

I think we have both fallen prey to trying to display code here and having some of it eaten or transformed. I just fixed mine, so take a look at it. Stefanefotso has included accommodation for the newline... which will not exist if your events are broken and is completely unnecessary. we're both saying, walk forward through the event, then when you get to the spot you want to grab... grab it.