topic problem with field extraction in Splunk Search

problem with field extraction

a212830 — Mon, 28 Sep 2020 11:22:16 GMT

Hi,

I'm trying to extract a field from a source, and when I test it, it appears to work, but in practice, it's grabbing too much.

Two examples are below. I want to create a "trapType" field, and report on it. It should contain "Vendor Specific" or "Authentication Failure". When I test it, everything looks ok, but then when I run a search, it's including everything after the values.

The field extract is: (?i) Type: (?P.+)

10:43:20.000 AM

Trap: 25311425
Thu Feb 9 10:43:20 2012
Src IP: 6.7.8.9
Agent IP: 6.7.8.9
Trap Type: Authentication Failure
Specific Type: 0
Enterprise: 1.3.6.1.6.3.1.1.5
Object:1.3.6.1.4.1.9.2.1.5.0 Value:6.7.8.9

Trap: 25311426
Thu Feb 9 10:43:20 2012
Src IP: 1.2.3.4
Agent IP: 1.2.3.4
Trap Type: Vendor Specific
Specific Type: 1
Enterprise: 1.3.6.1.4.1.9.9.41.2
Object:1.3.6.1.4.1.9.9.41.1.2.3.1.2.3175321 Value:PIM
Object:1.3.6.1.4.1.9.9.41.1.2.3.1.3.3175321 Value:5
Object:1.3.6.1.4.1.9.9.41.1.2.3.1.4.3175321 Value:INVALID_SRC_REG
Object:1.3.6.1.4.1.9.9.41.1.2.3.1.5.3175321 Value:Received Register from 1.2.3.4 for (1.2.3.4, 1.2.3.4), not willing to be RP
Object:1.3.6.1.4.1.9.9.41.1.2.3.1.6.3175321 Value:264d 04:40:27

Re: problem with field extraction

clintsharp — Thu, 09 Feb 2012 16:47:29 GMT

Can you either put up a pastebin of the input source or include in a code section so we can see the true raw input values? That would help me load it into a test instance to help diagnose the issue with your regex.

Re: problem with field extraction

joshd — Thu, 09 Feb 2012 17:28:41 GMT

Could you be a little more clear in what you are seeing, what do you mean you are including everything after the values? if the event you are inspecting is a true multiline event then you should have no issue with using that field extraction inside your props.conf ... you could also be a little more specific like so:

EXTRACT-fieldname = (?im).*Trap\sType:\s(?<fieldname>.+)

Your regex should match everything until the end of the line, so if anything else is being picked up thats why I ask if it's a true multiline event you're inspecting.

Re: problem with field extraction

a212830 — Thu, 09 Feb 2012 18:28:30 GMT

Yes, it's a multiline event. I've attached a screenshot of what happens.

Re: problem with field extraction

joshd — Thu, 09 Feb 2012 20:29:01 GMT

Unfortunately I cannot see the screenshot... check my updated example before, realized I forgot to add (?m) at the start to specify multiline events palmface

Re: problem with field extraction

a212830 — Thu, 09 Feb 2012 22:22:55 GMT

I tried the latest change and I'm still getting the same thing (bounced the app). I keep trying to upload the image, but it doesn't get added. Here's the text from the "pull-out" window for the trapType field. It's taking the information after the Trap Type: value, but it should stop at the end of that line, and it's not.

Authenticatio....6.1.4.1.3224 4,629 9.719%

Authenticatio....4.1.2021.251 1,440 3.023%

Authenticatio...e:10.47.89.25 1,060 2.226%

Authentication...e:10.93.65.11 788 1.654%

Authentication...4.1.564.101.1 500 1.05%

Authentication...10.165.27.145 480 1.008%

Authentication...e:10.47.89.25 444 0.932%

Authenticatio...1.8072.3.2.255 392 0.823%

Vendor Specif...1.4.1.3375.2.4 382 0.802%

Vendor Specif...1.4.1.3375.2.4

Re: problem with field extraction

clintsharp — Thu, 09 Feb 2012 22:23:47 GMT

I'll ask again, can you post a pastebin of the original log data? Something with the line breaks and everything preserved?

Re: problem with field extraction

a212830 — Mon, 28 Sep 2020 11:22:47 GMT

Not sure what a pastebin is, but I did a copy/paste of some of the log data.

Trap: 24910058
Thu Feb 9 02:01:11 2012
Src IP: 12.3.4.5
Agent IP: 1.2.3.4
Trap Type: Vendor Specific
Specific Type: 1
Enterprise: 1.3.6.1.4.1.9.9.43.2
Object:1.3.6.1.4.1.9.9.43.1.1.6.1.3.3997 Value:1
Object:1.3.6.1.4.1.9.9.43.1.1.6.1.4.3997 Value:4
Object:1.3.6.1.4.1.9.9.43.1.1.6.1.5.3997 Value:6

Trap: 24910059
Thu Feb 9 02:01:11 2012
Src IP: 1.2.3.4
Agent IP: 1.2.3.4
Trap Type: Vendor Specific
Specific Type: 1
Enterprise: 1.3.6.1.4.1.9.9.41.2
Object:1.3.6.1.4.1.9.9.41.1.2.3.1.2.0 Value:PAGP
Object:1.3.6.1.4.1.9.9.41.1.2.3.1.3.0 Value:6
Object:1.3.6.1.4.1.9.9.41.1.2.3.1.4.0 Value:PAGP
Object:1.3.6.1.4.1.9.9.41.1.2.3.1.5.0 Value:2012 Feb 09 02:01:11 eastern -05:00 %PAGP-5-PORTTOSTP:Port 2/17 joined bridge port 2/17
Object:1.3.6.1.4.1.9.9.41.1.2.3.1.6.0 Value:279122402

Trap: 24910060
Thu Feb 9 02:01:12 2012
Src IP: 1.2.3.4
Agent IP: 1.2.3.4
Trap Type: Vendor Specific
Specific Type: 1
Enterprise: 1.3.6.1.4.1.9.9.41.2
Object:1.3.6.1.4.1.9.9.41.1.2.3.1.2.61328708 Value:PIM
Object:1.3.6.1.4.1.9.9.41.1.2.3.1.3.61328708 Value:5
Object:1.3.6.1.4.1.9.9.41.1.2.3.1.4.61328708 Value:INVALID_SRC_REG
Object:1.3.6.1.4.1.9.9.41.1.2.3.1.5.61328708 Value:Received Register from 1.2.3.4 for (1.2.3.4, 2.3.4.5), not willing to be RP
Object:1.3.6.1.4.1.9.9.41.1.2.3.1.6.61328708 Value:468d 06:45:54

Re: problem with field extraction

clintsharp — Fri, 10 Feb 2012 17:44:31 GMT

Thanks, I now have the data, but I'm curious, can you provide what your inputs.conf and props.conf entries for this file and sourcetype look like? Without some changes, Splunk splits this data into multiple events etc, so I'm curious if you've already done some work to teach splunk how to interpret these events like setting LINE_BREAKER, or specifying regexes to teach splunk to break events.

BTW, your rex extraction works fine for me, so I'm still trying to dig in and reproduce your issue.