topic Why are field transformations not capturing from source field? in Splunk Search

Why are field transformations not capturing from source field?

thenino — Tue, 21 May 2019 18:20:05 GMT

I am trying to create a new field called collection which is extracted from the existing source field. I am able to extract the field during an adhoc search, but want to create it using the field transformations without having to generate a regex during each search.

The source field value is just a path (ex: source=D:\Logs\SomeCollectionName_SomeDomain\SomeDomain.log) and I am extracting part of that path to aggregate by collection. I want to specifically want to target the source field to regex not _raw.

BEFORE Extraction: source=D:\Logs\SomeCollectionName_SomeDomain\SomeDomain.log
AFTER Extraction: collection=SomeCollectionName

I have created a field transformation called testcollection and nothing is being extracted at search time.

Here are my settings for the testcollection field transformations (permissions are set for everyone to be able to read in the search app)

Maybe I have a misunderstanding of Field Transformations and should be using field extractions any guidance would be helpful. I am just using the default formatting, but maybe that is incorrect.

Note:

I have gone through the documentation for field transformations and field extracts. I understand how to extract new fields during a search, but I want this new field to be available to all of the users in our account.

Re: Why are field transformations not capturing from source field?

marycordova — Thu, 23 May 2019 18:31:40 GMT

Use and extraction for this instead of a transformation:

Re: Why are field transformations not capturing from source field?

thenino — Fri, 24 May 2019 14:48:44 GMT

Thank you. I tried that at first and I was running into this issue (I had to disable my json browser extension)

Your entry was not saved. The following error was reported: SyntaxError: Unexpected token < in JSON at position 0.

https://answers.splunk.com/answers/106487/your-entry-was-not-saved-the-following-error-was-r.html

Once I disabled my extension I was able to create the field extraction. I will report back if it works as expected.

Re: Why are field transformations not capturing from source field?

thenino — Wed, 30 Sep 2020 00:43:24 GMT

I used both field extractions and field transformations. The problem is _raw does not include the source field. I am able to parse anything that matches the regex in _raw but it is not parsing the actual source field. I want to be able to target the source field like I can during adhoc searches example:
source=D:\Logs\MyCollection_SomeDomain\SomeDomain.log

Some search
|rex field=source "(?<CollectionMap>\w+)\_"

Expected: CollectionMap=MyCollection
Actual: Anything with an underscore in _raw matches

Re: Why are field transformations not capturing from source field?

marycordova — Thu, 30 May 2019 04:39:13 GMT

Do you have access directly to the props.conf file instead of using the UI?

Re: Why are field transformations not capturing from source field?

thenino — Thu, 30 May 2019 15:43:12 GMT

We are using splunk cloud so no access to the props.conf

Re: Why are field transformations not capturing from source field?

thenino — Thu, 30 May 2019 15:46:45 GMT

I think I found my answer here:
https://answers.splunk.com/answers/149597/im-struggling-with-how-i-should-be-doing-inputs-and-also-props-transforms-etc-stuff-within-splunk-cloud.html

I will have to open a support ticket with splunk. Thanks for your assistance.