Why do my extractions not work?

[thomastaylor]

Hello everyone!

I have an event that looks like this (I omitted the sensitive information):

2018-06-07 09:55:16 ERROR ServiceRestCompany:285 - Error 500 in method: getServiceMethod3
Listener refused the connection with the following error:
ORA-55555, TNS:listener could not find available handler with matching protocol stack

I wrote a regular expression to extract "Service" and "Method3" in the line:

in method: get Service Method

Regular expression:
in method:\sget(?.+?(?=[A-Z]))(?.+?(?=\s))

Whenever I search for index="..." sourcetype="..." the events show WITH the fields extracted; however if I click on the extracted fields below the event to add to the search add, it refreshed the page and nothing shows up.

I do have suspicion of why this may be occurring. I have another sourcetype that also uses the field names of method and system in the same index. I intentionally named them the same because I want to be able to search method="blank" and it pull from both of the different source types in the index.

Any help would be appreciated!

[thomastaylor · Accepted solution]

Thank you to @yannk and @somesoni2 for providing answers to my question. I configured a work around that fits for my needs appropriately.

Scenario:

1. HTTP Event Logs contain a URI like this: /services/Service/Method in them
2. Java Traceback logs contain this in them: getServceMethod

They both come from different sourcetypes, but I wanted to unify them to where I could search index=main method=(insert method here). I configured the Regex and added a field named "method" for both sourcetypes. The peculiar part of this scenario is that when I searched for index=main method=(whatever here) it would only pull the HTTP event logs; however, if I searched index=main method!=(whatever here) it would actually pull both the Java and HTTP event logs that did not contain that method.

So, after working on this for a few hours, I cut my losses and just added tags to the method fields in both sourcetypes. I can now search index=main tag::method=(whatever here).

Thank you everyone!

[yannK]

if you suspect that another extraction is already creating a field named "method",
you can test by changing your field to "method2", and see if any of those 2 are extracted.

try to use a rex command inline to validate your extraction

mysearch  | rex "in method:\sget(?<service2>[A-Z]\w+)(?<method2>[A-Z]\w+)| table service* method* _raw

[thomastaylor]

@yaank I found the problem but not the solution. I cloned the sourcetype and indexed the data without the transformation I was using. After completing that action, everything works as normal. Do you know why this transformation: SEDCMD-truncate-stacktrace = s/\s+at\s[\s\S]+//g would affect this?

[yannK]

if you truncates the raw event, maybe the event displayed is not exactly as it in stored.
look at your field _raw to see that is the raw event.

[thomastaylor]

Splunk is giving some peculiar behavior. It does extract the fields correctly; however, when I add:

index=myindex system2=cascade

nothing shows up. "No results found". If I remove the system2=cascade, it shows me everything and I can verify that the specific sourcetype contains "system2 = Cascade" beneath the event.

Very odd, I'll be honest.

[yannK]

As you are extracting the field in the search itself with an inline rex, you cannot use the extracted fields yet to filter....
first get your extraction rights, then you can make it an automatic field extraction (instead of in line the search), and you will be able to use it as a base filter for that particular sourcetype.

[yannK]

A regex extraction is all or nothing, maybe the regex is wrong.
To troubleshoot, start with a regex to extract only 1 of the 2 fields, then work from there.

[thomastaylor]

If I completely get rid of my regular expression, and extract something else from the log and name it "method" it will work. (I used the Splunk regular expression highlight extractor)

Two options off the top of my head:

1. Maybe something wrong with the regex that I wrote?
2. If two of the same sourcetypes have the same field name and field value for the name, Splunk can potentially mess up and only display one sourcetype.

Any thoughts?

[somesoni2]

What else are you specifying in your search (replace sensitive data with #)?

[thomastaylor]

I am only specifying the index and sourcetype; however when I add the method to the search, that's when it suddenly doesn't find anything even though I verbatim added it from the event. It's just odd to me why this isn't working.

I was given instructions from @yannK to try and extract with a different name such as "method2". Unfortunately, this still yields the same result.