topic Re: How do I write the regex to extract these values from my sample data? in Splunk Search

How do I write the regex to extract these values from my sample data?

splunker1981 — Tue, 17 Nov 2015 02:30:36 GMT

Evening Splunk experts,

I am stuck trying to perform an extraction. I am using the built-in tool, but it keeps generating an incorrect regex. When I try to generate one myself, which works just fine using only regex tester, I get a message that no matches were found.

Below are sample logs from an in-house app and I am trying to create a field called gen_id and store the value to the right of the GENDID in the logs.

Nov 16 20:34:39 dtcSource01 dtc_data: detailStatus: Message finished GENID 399488 done
Nov 16 20:34:39 dtcSource01 dtc_data: detailStatus: MMR Message done RGENID 4 GENID 399488
Nov 16 20:34:39 dtcSource01 dtc_data: detailStatus: DSTC: shield GENID 399488
Nov 16 20:34:39 dtcSource01 dtc_data: detailStatus: MMR processing Begin RGENID 4 GENID 399488 to local shield
Nov 16 20:34:36 dtcSource01 dtc_data: detailStatus: GENID 399488 queued for processing
Nov 16 20:34:36 dtcSource01 dtc_data: detailStatus: DSTC: Delivering GENID 399488 to DSTC (skipping local)
Nov 16 20:34:36 dtcSource01 dtc_data: detailStatus: DSTC: Tagging GENID 399488 for shield from 192.x.x.x
Nov 16 20:19:14 dtcSource01 dtc_data: detailStatus: Begin GENID 568833 GID 289575
Nov 16 20:19:12 dtcSource01 dtc_data: detailStatus: GENID 568938 queued for local processing
Nov 16 20:19:12 dtcSource01 dtc_data: detailStatus: DSTC: Delivering GENID 568938 to DSTC (skipping work queue)
Nov 16 20:21:05 dtcSource01 dtc_data: detailStatus: GENID 398845 SubStatus 'working'
Nov 16 20:21:05 dtcSource01 dtc_data: detailStatus: DSTC: Delivering GENID 398845 to DSTC (skipping work queue)
Nov 16 20:21:05 dtcSource01 dtc_data: detailStatus: GENID 398845 queued for processing
Nov 16 20:19:17 dtcSource01 dtc_data: detailStatus: MMR processing Begin RGENID 4 GENID 568938 to local
Nov 16 20:19:14 dtcSource01 dtc_data: detailStatus: DSTC: Delivering GENID 568833 to DSTC (skipping local)
Nov 16 20:19:11 dtcSource01 dtc_data: detailStatus: DSTC: Delivering GENID 568899 to DSTC (skipping local)  
Nov 16 20:11:33 dtcSource01 dtc_data: detailStatus: DSTC: Tagging GENID 703728 for shield from 192.x.x.x

I tried the following commands, but they either don't capture what I need or capture data before or after the ID - not sure what I am doing wrong. A simple GENID\s\d+ should do it.

  | rex field=_raw "GENID (?<GENID>.*)"
  | rex field=_raw "GENID (?<GENID>\s\d+)"
  | rex field=_raw "GENID (?<GENID> \d+).*?"
  | (?P<FIELDNAME>\w+\s+\d+)

What I would like to have extracted are the following values - I want to be able to create a field called gen_id and store the values below.

    Fieldname         ID
    GENID            399488
    GENID            568833
    GENID            398845
    GENID            568833

Thanks for helping out.

Re: How do I write the regex to extract these values from my sample data?

sundareshr — Tue, 17 Nov 2015 03:00:29 GMT

Here's one way

GENID\s(?<gen_id>\d+)

Re: How do I write the regex to extract these values from my sample data?

splunker1981 — Tue, 17 Nov 2015 03:16:51 GMT

Thanks for the reply, but I don't follow. I tried adding that to the regex generator to test and I get an error "Invalid regex: unexpected end of pattern"

Re: How do I write the regex to extract these values from my sample data?

sundareshr — Tue, 17 Nov 2015 03:34:29 GMT

That's strange. It missing the field name. Try this GENID\s(?\d+)

Re: How do I write the regex to extract these values from my sample data?

sundareshr — Tue, 17 Nov 2015 03:35:22 GMT

It's missing again. For some reason, the field name is getting dropped here. In the online documentation (http://docs.splunk.com/Documentation/Splunk/6.0/SearchReference/rex) for rex. look at the first example. There a "from" after the ? within angle brackets. That's what is missing. You need to add that, you may change "from" to whatever you would like to name the field. Hope this explains it

Re: How do I write the regex to extract these values from my sample data?

yulianaif — Tue, 17 Nov 2015 08:16:08 GMT

I hope this work. I use \s before GENID because there is value "RGENID" in your data.

| rex field=_raw "\sGENID\s(?<gen_id>\d+)"

Re: How do I write the regex to extract these values from my sample data?

ppablo — Wed, 18 Nov 2015 02:35:58 GMT

Hi @sundareshr

I just edited your original answer to get the field name within angle brackets to show up properly. Looks like it has to be within a Code Sample box on its own line. I've filed a bug with angle brackets not being rendered properly on this site when only surrounded by backticks.

Cheers!

Patrick

Re: How do I write the regex to extract these values from my sample data?

sundareshr — Wed, 18 Nov 2015 03:10:43 GMT

@ppablo Thanks. I tried both backticks as well as code sample box on its own line. I have not been able to get those angle brackets to show these past couple days.