Security

Issues with field extraction possibly after splunk upgrade from 6.4.2 to 6.5.1

usha_nittala
New Member

I am unable to extract any field either using event actions on UI or using transforms.
Recently I have upgraded splunk from 6.4.2 to 6.5.1 and after that seeing weird issue with field extraction.

Sample log file :

123.123.123.119 -   -   [16/Mar/2017:06:30:26 -0400]    302 231 "https://mytest.test.com/test/recommetest/viewtest" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) applert/345.36 (KHTML, like Gecko) Chrome/safs223 Safari/517.36"  GET "/test.ico" ""  HTTP/1.1    5238    -   -   -   REQ45678765432223459876543  101.111.12.111

I am using below props and transforms:

props.conf :

[my_access]
REPORT-web-log = web-log
REPORT-webstats-fields = webstats-fields
EXTRACT-x_client_ip = ^(?P<x_client_ip>\d+\.\d+\.\d+\.\d+)\t+

transforms.conf

[webstats-fields]
REGEX = (?<x_client_ip>.*?)\t-\t-\t.*?\t(?<http_status>.*?)\t(?<content_length>.*?)\t(?<referer>.*?)\t(?<user_agent>.*?)\t(?<method>.*?)\t(?<uri>.*?)\t(?<query_string>.*?)\t(?<protocol>.*?)\t(?<duration>.*?)\t-\t-\t-\t(?<id>.*)\t(?<client_ip>\d+\.\d+\.\d+\.\d+)\t


[web-log]
REGEX = (?<x_client_ip>\S+).*?(?<http_status>[0-9]{3})\s+(?<content_length>\S+)\s+"(?<referer>\S+)"\s+"(?<user_agent>.*?)"\s+(?<method>\S+)\s+"(?<uri>\S+)"\s+"(?<query_string>\S*)"\s+(?<protocol>\S+)\s+(?<duration>\S+)\s+"(?<user_detail>\S+)"\s+(?<client_info>\S+)\s+(?<plan_info>\S+)\s+(?<id>\S+)\s+(?<client_ip>\S+)\s+(?<am_info>.*)

I tested these regex in regex101.com but still when I applied it , I could not see extracted fields on left side under interesting fields.
I tried creting transforms both on UI and giving permissins and by keeping these files under local directory of app on servers.

I even tried extracting single field using EXTRact clause in props.conf but to no use.

Before upgrading splunk I never faced issue with field extraction but I now i am not able to extract field. I am not sure if its related to upgrade or anything wrong with my configuration.

Appreciate any help in this regard.

Thanks.

0 Karma

sloshburch
Splunk Employee
Splunk Employee

BTW: Are field extractions not working for ANYTHING? Or just this sourcetype? What's the scope of the issue? It should inform the issue better.

0 Karma

sloshburch
Splunk Employee
Splunk Employee

EXTRACT-x_client_ip is being defined in three places: the props and the two transforms stanzas. I recommend pulling a sample of the data along with the config to your local dev env and try toggling comments for them until you can determine which one is not working.

Second debugging trick is to comment out the transforms and do the field extractions in a rex command. If it still doesn't work in a rex command then chop out half and see if the remaining half works. If it does, then toggle to the other half and try that. Divide and conquer until you identify the portion causing an issue.

Lastly, you might want to open a support ticket if this was working previously but not anymore.

0 Karma

woodcock
Esteemed Legend

A field can only be interesting if it occurs in at least 90X% (is it 95?) of all events in the returned results. The way to add it to the fields sidebar if it is NOT interesting is to add it to the Selected Fields list:
Click All Fields.
The Select Fields dialog box shows a list of fields in your events and ALL fields will be shown.
The # of Values column shows the number of unique values for each field in the events.
Search for your field name and click the checkbox next to it.
Click save.

You can also click the > icon icon next to your event under the i header on the events tab to turn it into a v and this will show you ALL fields for that event, even the ones that are not interesting.

cmeerbeek
Path Finder

Did this work before?

Please be aware of app-context and permissions. If you define props and transforms in one app and want to search the data in the other app these settings need to be set to global.

0 Karma

usha_nittala
New Member

And when I tried to use extract field option on the UI, It highlighted the field I was trying to extract alt text

0 Karma

usha_nittala
New Member

Hi cmeerbeek ,

This is working for other source of logfile with similar format.
I have placed these transforms and props under /etc/apps/search/local and checked metadata for the same for permissions.

When I run splunk cmd btool props list and transforms list , I see these entries but I don't understand why I am not able to see underinteresting fields.

0 Karma

cmeerbeek
Path Finder

OK clear. No permissions error than.

Than there must be an issue with the regex. Can you share the regex101 you've created? Press save on the left and they will generate an URL which you can share.

0 Karma

usha_nittala
New Member

Thanks for helping me out cmeerbeek .
Here is the link : - https://regex101.com/r/b0K6bh/1

0 Karma

cmeerbeek
Path Finder

The regex works it seems but you cannot be 100% sure that tabs and spaces are the same in Splunk... Did you tried the same regex with the rex command? What is the result than?

Small tip; try to be more specific in your regex. If you change http_status from .*? to \d{3} performance will be much better.

0 Karma

rjthibod
Champion

Are you running your searches in the search dashboard in "verbose mode"? On the right-hand side of the search dashboard under the timepicker is a selector that determines the search mode: Fast Mode, Smart Mode, and Verbose Mode.

When trying to check field extractions, start out in Verbose mode.

If that doesn't work, trying running btool from the command-line to double check if there are any errors.

$ <SPLUNK_HOME>/bin/splunk btool check

or this for more debug output

$ <SPLUNK_HOME>/bin/splunk btool check --debug

0 Karma

usha_nittala
New Member

Thanks for the reply rjthibod .
I ran this command now and it checked all .conf files on the servers and did not throw any error.

I also checked in verbose mode but to no avail.

0 Karma