topic Re: Why do my extractions not work? in Splunk Search

Why do my extractions not work?

thomastaylor — Mon, 18 Jun 2018 20:32:00 GMT

Hello everyone!

I have an event that looks like this (I omitted the sensitive information):

2018-06-07 09:55:16 ERROR ServiceRestCompany:285 - Error 500 in method: getServiceMethod3
Listener refused the connection with the following error:
ORA-55555, TNS:listener could not find available handler with matching protocol stack

I wrote a regular expression to extract "Service" and "Method3" in the line:

in method: get Service Method

Regular expression:
in method:\sget(?.+?(?=[A-Z]))(?.+?(?=\s))

Whenever I search for index="..." sourcetype="..." the events show WITH the fields extracted; however if I click on the extracted fields below the event to add to the search add, it refreshed the page and nothing shows up.

I do have suspicion of why this may be occurring. I have another sourcetype that also uses the field names of method and system in the same index. I intentionally named them the same because I want to be able to search method="blank" and it pull from both of the different source types in the index.

Any help would be appreciated!

Re: Why do my extractions not work?

somesoni2 — Mon, 18 Jun 2018 21:10:34 GMT

What else are you specifying in your search (replace sensitive data with #)?

Re: Why do my extractions not work?

yannK — Tue, 19 Jun 2018 00:00:28 GMT

if you suspect that another extraction is already creating a field named "method",
you can test by changing your field to "method2", and see if any of those 2 are extracted.

try to use a rex command inline to validate your extraction

mysearch  | rex "in method:\sget(?<service2>[A-Z]\w+)(?<method2>[A-Z]\w+)| table service* method* _raw

Re: Why do my extractions not work?

thomastaylor — Tue, 19 Jun 2018 12:11:35 GMT

Splunk is giving some peculiar behavior. It does extract the fields correctly; however, when I add:

index=myindex system2=cascade

nothing shows up. "No results found". If I remove the system2=cascade, it shows me everything and I can verify that the specific sourcetype contains "system2 = Cascade" beneath the event.

Very odd, I'll be honest.

Re: Why do my extractions not work?

thomastaylor — Tue, 19 Jun 2018 12:15:03 GMT

I am only specifying the index and sourcetype; however when I add the method to the search, that's when it suddenly doesn't find anything even though I verbatim added it from the event. It's just odd to me why this isn't working.

I was given instructions from @yannK to try and extract with a different name such as "method2". Unfortunately, this still yields the same result.

Re: Why do my extractions not work?

thomastaylor — Tue, 19 Jun 2018 12:27:39 GMT

If I completely get rid of my regular expression, and extract something else from the log and name it "method" it will work. (I used the Splunk regular expression highlight extractor)

Two options off the top of my head:

Maybe something wrong with the regex that I wrote?
If two of the same sourcetypes have the same field name and field value for the name, Splunk can potentially mess up and only display one sourcetype.

Any thoughts?

Re: Why do my extractions not work?

thomastaylor — Tue, 19 Jun 2018 14:26:02 GMT

@yaank I found the problem but not the solution. I cloned the sourcetype and indexed the data without the transformation I was using. After completing that action, everything works as normal. Do you know why this transformation: SEDCMD-truncate-stacktrace = s/\s+at\s[\s\S]+//g would affect this?

Re: Why do my extractions not work?

yannK — Tue, 19 Jun 2018 16:34:43 GMT

A regex extraction is all or nothing, maybe the regex is wrong.
To troubleshoot, start with a regex to extract only 1 of the 2 fields, then work from there.

Re: Why do my extractions not work?

yannK — Tue, 19 Jun 2018 16:36:56 GMT

if you truncates the raw event, maybe the event displayed is not exactly as it in stored.
look at your field _raw to see that is the raw event.

Re: Why do my extractions not work?

yannK — Tue, 19 Jun 2018 16:37:41 GMT

As you are extracting the field in the search itself with an inline rex, you cannot use the extracted fields yet to filter....
first get your extraction rights, then you can make it an automatic field extraction (instead of in line the search), and you will be able to use it as a base filter for that particular sourcetype.

Re: Why do my extractions not work?

thomastaylor — Fri, 22 Jun 2018 12:37:25 GMT

Thank you to @yannk and @somesoni2 for providing answers to my question. I configured a work around that fits for my needs appropriately.

Scenario:

HTTP Event Logs contain a URI like this: /services/Service/Method in them
Java Traceback logs contain this in them: getServceMethod

They both come from different sourcetypes, but I wanted to unify them to where I could search index=main method=(insert method here). I configured the Regex and added a field named "method" for both sourcetypes. The peculiar part of this scenario is that when I searched for index=main method=(whatever here) it would only pull the HTTP event logs; however, if I searched index=main method!=(whatever here) it would actually pull both the Java and HTTP event logs that did not contain that method.

So, after working on this for a few hours, I cut my losses and just added tags to the method fields in both sourcetypes. I can now search index=main tag::method=(whatever here).

Thank you everyone!