topic Re: Why is my field extraction not consistent across all events? in Splunk Search

Why is my field extraction not consistent across all events?

diavolo — Wed, 01 Feb 2017 16:05:08 GMT

I want to extract a field which is uuid format and name it instanceid.

props.conf settings

EXTRACT-fields_5 = \[[i]nstance:\s+(?P<instanceid>[0-9a-f]{8}\-[0-9a-f]{4}\-[0-9a-f]{4}\-[0-9a-f]{4}\-[0-9a-f]{12})

For logs like ...

2017-01-01 00:00:00.000 99999 INFO xxxxxxxxxxxx [-] [instance: 01234567-89ab-cdef-0123-456789abcdef] Instance destroyed successfully.

However, it works for some events but it doesn't for some other events.
When I changed the field name to nstanceid or istanceid in regex, it works for all events. I don't know what's wrong with the field name instanceid.
OTOH, rex command with above regex (field name is instanceid) works well.

Would somebody give me the reason why??

Re: Why is my field extraction not consistent across all events?

horsefez — Wed, 01 Feb 2017 16:20:03 GMT

Hi diavolo,

try the following.

(?:\[instance:\s+)(?P<instanceid>[0-9a-f]{8}\-[0-9a-f]{4}\-[0-9a-f]{4}\-[0-9a-f]{4}\-[0-9a-f]{12})(?:\])

Should work fine now.

Re: Why is my field extraction not consistent across all events?

diavolo — Wed, 01 Feb 2017 16:49:16 GMT

Unfortunately, it doesn't work. The field can't be extracted in some events.

Re: Why is my field extraction not consistent across all events?

DalJeanis — Wed, 01 Feb 2017 17:36:28 GMT

1) when you say "change the field name" are you talking about the underlying data, or the field name being extracted by the regex?
2) can you post an example of an event that the extract did NOT work for?

Re: Why is my field extraction not consistent across all events?

diavolo — Tue, 29 Sep 2020 12:41:51 GMT

1) The latter one. I changed regex from (?P<instanceid>...) to (?P<nstanceid>...). It worked.
2)
- Worked:
2017-01-06 03:08:35.416 21995 INFO nova.virt.libvirt.driver [-] [instance: 40624b9c-8179-4cb0-82ec-924ee5362cc0] Instance destroyed successfully.
- Not Worked:
2017-01-06 03:07:25.932 21995 DEBUG nova.network.neutronv2.api [-] [instance: 6708c71b-0f49-4b0b-8040-fec13e3e2a4b] get_instance_nw_info() _get_instance_nw_info /usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py:602

Re: Why is my field extraction not consistent across all events?

DalJeanis — Wed, 01 Feb 2017 18:58:03 GMT

The problem may be the (?P at the beginning of the regex.

Also, I believe you can shorthand hex digits as \h, so your regex can look a bit cleaner if you try this -

 EXTRACT-fields_5 = \[instance:\s+(?<instanceid>\h{8}\-\h{4}\-\h{4}\-\h{4}\-\h{12})

see this page for more details - http://www.regular-expressions.info/refext.html

Re: Why is my field extraction not consistent across all events?

MuS — Wed, 01 Feb 2017 19:52:37 GMT

Hi diavolo,

my guess would be that in some events there is actually a field called instanceid.
Try to use a completely new/different field name to test your field extraction, something like this should work for you:

 \[instance:\s+(?<ThisIsMyTestFieldName>[^\]]+)

cheers, MuS

Re: Why is my field extraction not consistent across all events?

diavolo — Thu, 02 Feb 2017 03:02:19 GMT

Thanks MuS,
instanceid is not used anywhere. Changing field name like instance_id works fine. But I was wondering why...

Re: Why is my field extraction not consistent across all events?

diavolo — Thu, 02 Feb 2017 04:16:47 GMT

? didn't fix the problem... Also, \h for hex didn't work.

Re: Why is my field extraction not consistent across all events?

diavolo — Thu, 02 Feb 2017 05:12:47 GMT

Mmm... After I changed the extracted field name in regex from instanceid to instance_id for workaround, it doesn't work for some events. It worked fine soon after I did change, but 1 hour later, it doesn't.

Re: Why is my field extraction not consistent across all events?

horsefez — Thu, 02 Feb 2017 14:51:14 GMT

Could you provide us with the exakt event _raw payload that doesn't match this regex?

Re: Why is my field extraction not consistent across all events?

woodcock — Thu, 02 Mar 2017 19:56:31 GMT

The problem is two-fold: either the event does not have what you think all of them does (non-conforming event data) OR your RegEx is slightly off and does not fully accommodate all variations of the events (insufficient RegEx). In either case, here is what you need to do to figure it out. Deploy the version that works best, let's say that you are using a field name of instance_id. Then run a search like this:

... NOT instance_id="*"

This will show you all events that do not have a field called instance_id. You adjust your RegEx or ignore that type of event (by putting an exclusion for it in your base search) and keep repeating this cycle until you have no events returned from that search.