topic Re: Field Extraction issue in Splunk Search

Field Extraction issue

rajgowd1 — Tue, 29 Sep 2020 11:20:09 GMT

HI Experts,
i am able to exact 4th and 5th fields from below log but i am able to exact get the value if the 4th or 5th filed is HOSTNAME but if it is IPaddress then i am not able retrieve.here is the sample log

tcp 0 0 10.40.88.178:7171 10.40.88.175:50326 ESTABLISHED
tcp 0 0 12b8-splfwd02.ndm.nsro:7171 10.40.88.170:50326 TIME_WAIT
tcp 0 0 10.40.88.178:7171 poc-card-luna2.nad.ns:50326 TIME_WAIT
tcp 0 0 12b8-splfwd02.ndm.nsro:7171 10.30.88.145:50326 ESTABLISHED
tcp 0 0 10.40.88.178:7171 poc-card-luna4.nad.ns:50326 SYNC_SENT
tcp 0 0 12b8-splfwd03.ndm.nsro:7171 poc-card-luna1.nad.ns:46756 TIME_WAIT
tcp 0 0 12b8-splfwd03.ndm.nsro:7171 10.30.88.117:46756 SYNC_SENT
tcp 0 0 10.40.88.178:7171 poc-card-luna2.nad.ns:46756 TIME_WAIT
tcp 0 0 12b8-splfwd03.ndm.nsro:7171 10.40.83.157:46756 TIME_WAIT
tcp 0 0 12b8-splfwd03.ndm.nsro:7171 poc-card-luna4.nad.ns:46756 ESTABLISHED
tcp 0 0 12b8-splfwd04.ndm.nsro:7171 poc-card-luna1.nad.ns:46756 TIME_WAIT
tcp 0 0 10.40.88.178:7171 poc-card-luna3.nad.ns:46756 SYNC_SENT
tcp 0 0 12b8-splfwd04.ndm.nsro:7171 poc-card-luna2.nad.ns:46756 SYNC_SENT
tcp 0 0 12b8-splfwd04.ndm.nsro:7171 poc-card-luna4.nad.ns:46756 ESTABLISHED
tcp 0 0 12b8-splfwd04.ndm.nsro:7171 poc-card-luna2.nad.ns:46756 ESTABLISHED

Re: Field Extraction issue

richgalloway — Sat, 08 Oct 2016 19:49:47 GMT

How are you extracting the fields? Is this a search-time or index-time extraction?

Re: Field Extraction issue

sundareshr — Sat, 08 Oct 2016 20:02:20 GMT

You have few options.

1) Edit props/transforms per instructions in the blog post. Your data appears to be space delimited, so you will specific " " in your transforms instead of ","

http://blogs.splunk.com/2013/03/11/quick-n-dirty-delimited-data-sourcetypes-and-you/

2) The other option is to use Interactive Field Extractor (IFX) to extract the fields as space delim. IFX will generate the similar props/transforms edits for the extractions.

3) Use regex in your SPL like this

base search | rex "(?<f1>[^\s]+)\s(?<f2>[^\s]+)\s(?<f3>[^\s]+)\s(?<f4>[^\s]+)\s(?<f5>[^\s]+)\s(?<f6>[^\s]+)" | table f*

Re: Field Extraction issue

rajgowd1 — Sat, 08 Oct 2016 20:02:48 GMT

Hi Rich,thanks for your response.
From logs,I am trying to extract using splunk UI.
I think I am able extract multiple fields.

Re: Field Extraction issue

rajgowd1 — Sat, 08 Oct 2016 20:02:48 GMT

Hi Rich,thanks for your response.
From logs,I am trying to extract using splunk UI.
I think I am able extract multiple fields.

Re: Field Extraction issue

govindsinghrawa — Tue, 29 Sep 2020 11:19:24 GMT

Below Regex can be used in field extractor in write your own regular expression syntax:



^(?< protocol>[\S]+)\s*?(?< field1>[\S]+)\s*?(?< field2>[\S]+)\s*?(?< hostOrIp1>[^:]+):(?< hostOrIp1Port>[\S]+)\s*?(?< hostOrIp2>[^:]+):(?< hostOrIp2Port>[\S]+)\s.*?(?< state>[\S]+)

During search time this regex can also be used with rex to extract fields if fields are not already extracted:



yourBaseSearch

|rex field=_raw "^(?< protocol>[\S]+)\s*?(?< field1>[\S]+)\s*?(?< field2>[\S]+)\s*?(?< hostOrIp1>[^:]+):(?< hostOrIp1Port>[\S]+)\s*?(?< hostOrIp2>[^:]+):(?< hostOrIp2Port>[\S]+)\s.*?(?< state>[\S]+)"

| table hostOrIp1, hostOrIp2

NOTE: Please remove the space in each of the tags, example "< protocol>", "< field1>" etc.

Re: Field Extraction issue

richgalloway — Sat, 08 Oct 2016 20:17:10 GMT

If you're using the Splunk UI you're probably using the automatic field extractor which is most likely producing a regex string that is too specific. Try this regex, instead. It skips the first three space-delimited fields and extracts the next two space-delimited fields as 'src' and 'dest', respectively.

(?:[^\s]+\s){3}(?P<src>[^\s]+)\s(?<dest>[^\s]+)

Perhaps you want the 4th and 5th fields to be the host name/IP address and port. That regex string is

(?:[^ ]+ ){3}(?P<src>[^:]+):(?<dest>\d+)

Re: Field Extraction issue

rajgowd1 — Tue, 29 Sep 2020 11:20:18 GMT

Thank you govindsinghrawat.i am trying to create visualization by using all your experts answers.
is there way to create visualization by using similar search

both are giving same results

index=netstat|rex field=_raw "^(?[\S]+)\s*?(?[\S]+)\s*?(?[\S]+)\s*?(?[^:]+):(?[\S]+)\s*?(?[^:]+):(?[\S]+)\s.*?(?[\S]+)"
| eventstats count by state hostOrIp1 hostOrIp2 |eval CountConnStatus=count+" - "+state|chart values(CountConnStatus) by hostOrIp1 hostOrIp2

index=netstat |rex "(?[a-zA-Z0-9][^\s]+)\s(?[^\s]+)\s(?.*)" |eventstats count by CONN_STATUS CBOL_SERVER HSM |eval CountConnStatus=count+" - "+CONN_STATUS|chart values(CountConnStatus) by CBOL_SERVER HSM

Re: Field Extraction issue

govindsinghrawa — Sat, 15 Oct 2016 06:00:03 GMT

Hi @rajgowd

The above regex which I gave is to extract fields. Once the fields are extracted then you can use them i your commands.

Now coming to visulaizations. Visualizations can only be created depending on how you end your search, for example :

if you end your search with timechart command then you will immediately have visualization options for line, bar etch. Example:
yourBaseSearch | timechart count by yourField

If you have statistics being generated as aggregations like min, max, sum etc you can plot them using timechart, chart etc.