topic Re: extracting search fields using regexp in transforms.conf not working in Splunk Search

extracting search fields using regexp in transforms.conf not working

builder — Mon, 28 Sep 2020 09:40:26 GMT

I refered to the following documentation to try and get this working:

http://www.splunk.com/base/Documentation/3.0.2/Admin/CreateAdditionalSearchFields

I have a standard rails log file where every new request starts with a line like:

Started (GET|POST) "/some-path"

I want to extract "/some-path" into a field rails-action-path. I added the following to $SPLUNK_HOME/etc/system/local/transforms.conf (note that the documentation says to add it to $SPLUNK_HOME/etc/bundles/local/transforms.conf, but there is no bundles directory in my installation and my other conf files have all been in the system directory).

[get-rails-action-path]
REGEX = Started \S ([\S\/"\.]+)
FORMAT = $0 rails-action-path::$1
DEST_KEY = _meta

My sourcetype stanza for the rails logs in $SPLUNK_HOME/etc/apps/search/local/props.conf is as follows (again note, my props.conf has always lived here although the documentation refers to $SPLUNK_HOME/etc/bundles/local/props.conf).

[(?:::){0}*rails]
LINE_BREAKER = ([\r\n]).* [\r\n]+Started (POST|GET)
TRANSFORMS-rap = rails-action-path

Note that the last line was the only part I just added. The wildcard source type for the rails logs and the LINE_BREAKER have been there for awhile and behave as I expect.

After restarting my indexer with these changes, I do not see rails-action-path in the field list.

Any ideas?

Re: extracting search fields using regexp in transforms.conf not working

southeringtonp — Tue, 14 Jun 2011 22:34:19 GMT

You're looking at pretty old documentation. You're not really using Splunk version 3.0.2, right? All of the "bundles" stuff went out a long time ago and was replaced by "apps". Start here for the documentation, or use the "product version" dropdown in the upper left to make sure you're looking at something current.

Putting it in etc/system/local is alright, though the more common place in Splunk 4.x would be in /etc/apps/search/local, or in a new app of your own creation.

Suggestions / Issues:

The regex is a little off. You need \S+ (note the plus sign) to match more than one character.
You can leave out the $0
Use REPORT instead of TRANSFORMS for search-time extraction, and make sure that the name referenced in props.conf matches the stanza name used in transforms.conf.
DEST_KEY isn't needed

With those things in mind, try this:

#transforms.conf
[get-rails-action-path]
REGEX = Started \S+ ([\S\/"\.]+)
FORMAT = rails-action-path::$1

#props.conf
[(?:::){0}*rails]
LINE_BREAKER = ([\r\n]).* [\r\n]+Started (POST|GET)
REPORT-rap = get-rails-action-path

(Assuming your stanza name in props.conf is ok, since you say it's working for line breaker.)

Re: extracting search fields using regexp in transforms.conf not working

builder — Tue, 14 Jun 2011 23:33:43 GMT

I'm going to have to reply in two comments since it has a comment limit.

You are correct, I am using 4.2.1. I didn't notice the documentation was old. Sorry about that.

The missing + on the regexp and the non-matching stanza name were both just brain farts on my part, but I probably wouldn't have caught them after staring at this for so long.

Re: extracting search fields using regexp in transforms.conf not working

builder — Mon, 28 Sep 2020 09:40:31 GMT

I updated the conf files per the above and restarted my indexer. Now, rails-action-path shows up on my summary page as a field with properly parsed values. If I click on the field and select 'show only events with this field', it does the right thing (searching with sourcetype="space-rails" rails_action_path="*" and showing numerous events). However, if I select any specific listed value for the field (e.g., searching with sourcetype="space-rails" rails_action_path="/client_api/updateGameData"), it returns 0 events and says "Waiting for data" indefinitely.

What could be going on here?

Re: extracting search fields using regexp in transforms.conf not working

builder — Wed, 15 Jun 2011 00:43:23 GMT

So, I was reading over the 4.2.1 docs, per your advisement, and it sounds like the recommendation is actually search-time field extraction. Thus, I have reverted all the changes mentioned in this thread to my transform.conf and props.conf on my indexer machines. I have instead set up search-time extraction in my props.conf, but it's not working either (the field is back to not showing up). I will start a new thread for that.

Re: extracting search fields using regexp in transforms.conf not working

southeringtonp — Wed, 15 Jun 2011 01:18:43 GMT

Using REPORT as above, it will be a search-time transaction. DEST_KEY shouldn't be in there though; I copied and pasted from your example and failed to remove it until just now.

Re: extracting search fields using regexp in transforms.conf not working

southeringtonp — Wed, 15 Jun 2011 01:20:21 GMT

Not sure why it wouldn't show up and say "waiting for events", unless having DEST_KEY in there was interfering with things, which is quite possible.

Re: extracting search fields using regexp in transforms.conf not working

builder — Mon, 28 Sep 2020 09:40:42 GMT

I was using the instructions here for search time extraction and EXTRACT seemed simpler:

http://www.splunk.com/base/Documentation/4.2.1/Knowledge/Createandmaintainsearch-timefieldextractionsthroughconfigurationfiles

Also, the original confs were on my indexer. The EXTRACT is in props.conf on my search-head. The field is showing up using this method, but it didn't solve the problem. It's pretty much the same as before. I see the field and all extracted values. If I search by rails_action_path="*", I get all results, but any specific value returns 0 results/No matching events found.

Re: extracting search fields using regexp in transforms.conf not working

builder — Thu, 16 Jun 2011 17:44:33 GMT

Just going to start a new thread as this one seems to have died. : P