Why aren't all my field extractions working?

[ebs]

Hi,

I'm having an odd issue. I made some field extractions and validated them through Regex101. However only some of the fields are being extracted, not all. Initially they all work and then some disappear. Its a single Regex string so if there were any issues I don't know why other fields would be extracting but not others. And the sourcetype has not changed.

Does anyone have a solution for this or any inkling of what might be going on?

For reference here's my regex:

"log":\s"(?<log_source>[^\s]+)\s(?<ISO8601>[^\s+]+)\s+(?<log_level>[^\]]+)\s\[(?<exchangeId>[^\]]+)\]\s(?<RuleType>[^\.]+)\.\[(?<RuleName>[^\]]+)\]\s-\s(?<http_method>[^\|]+)\|(?<site>[^\|]+)\|(?<uri_path>[^\s\?"|]++)\|(?<status>[^\|]+)\|{\\"error_description\\":\\"(?<error_description>[^"]+)\\\",\\"error\\":\\"(?<error>[^\\]+)\\"}\\n

Log:

"log": "/opt/instance/log/access.log 2021-09-01T14:40:17,493 WARN [wUJHboi800nOHINLKnugbF1rBkcQ] Rule.[ErrorCapture] - POST|site.com|/oauth|400|{\"error_description\":\"Authorization code is invalid or expired.\",\"error\":\"invalid_grant\"}\n"

And it seems to only be the site field not extracting for whatever reason

[PickleRick]

For log_level you're trying to capture everything that's not a closing bracket. Are you sure that's what you want? Because I'm not 😉

In uri_path you have two pluses and you might stop capturing short of the pipe character (and you don't need to escape question mark in the character class here).

Curly brackets in the last part should be escaped, otherwise they are interpreted as repetition counters.

Finally, you're capturing error_description with the backslash escaping the quotation marrk, and the error without it (and you'll have problems if any of these fields contains those fields further escapes).

[ebs]

Hi PickleRick (love the name btw),

Thanks for improving my regex, honestly had not noticed that bracket one, although am a bit lost on your last point. Not exactly sure what you mean.

Anyway besides that, although you've improved my regex (again thank you) it doesn't help me with this particular issue which is my site field being absent in search despite being validated by Regex101 and being confirmed as extracted in Event Actions > Extract Fields in Splunk. When I try to search on it, its not there, nor is is it in the fields sidebar.

Do you know why this could be?

[PickleRick]

OK, So you defined extraction by using Event Action -> Extract Fields, you defined the regex, checked it further down the process, saved it but now it "won't work" for the events you want it to parse, right?

Two quick questions:

1. Are the events you're trying to parse of the same sourcetype you created the extraction for?

2. Do you use for searching the same user you defined the extractions with? (or did you share the extractions for other apps)

[ebs]

Hi, 

As stated in my original post, the sourcetype has not changed and the extractions have Global permissions, but nevertheless is searched with the same account that created them.

Also, the other fields in the regex work fine, it is only one missing. The site field, the rest appear as expected.

[ebs]

Okay, its getting worse. I just extracted a new log format, sourcetype is accurate, permissions are global and despite confirming its extracted in Event Actions > Extract Fields the fields aren't showing at all in the log dropdown

[ebs · Accepted solution]

Okay, I worked it out somehow. Because the field I was naming site, was an field alias it wasn't applying for some reason?

So now that I've named it the field name that its aliasing it works. No idea why, but that's what worked.