Field extraction will not go away unable to find in manager or props.conf

[jericksonpf]

Hi,
I have a field called UserID appearing in my searches that in two of my sourcetypes within the same index. Ive scoured the GUI manager looking for UserID and it is nowhere to be found, i have checked every props.conf file i can find on the box itself and i dont see that field extraction anywhere. I have tried restarting the server but the field keeps popping up. I want very desperately to edit that regex to make it work properly but I cant find it to edit it. Can anybody help? Is there a secret place where field type regexes are kept in splunk?

[jericksonpf · Accepted solution]

So since scouring the props and transforms files was not really working I decided to try a more blunt approach. I started disabling apps one at a time starting with the most recently installed.
Luckily the last two apps were not essential and it looks like the *nix app was the culprit.

I am not super spun up on how apps interact with each other but there was nothing in the props.conf file for the *nix app that would indicate that it would be preforming any extracts that pulled user ids on my EHR sourcetype. I also have the *nix app looking at its own unique index.

Shrugs at least it is gone now

[lguinn2]

So there are extractions in the *nix app that have been designated as system (which means global). This would cause the fields to appear in other apps, not just *nix. The local.meta file controls the visibility of field extractions and other knowledge objects.

That's how this interaction occurred, I'll bet. When you look at knowledge objects using the Manager UI, take a good look at the App column - this will tell you where to find the relevant configuration file.

I also think that Kristian's suggestion is a good - fields can be extracted based on many criteria, not just sourcetype.

[alanfinlay]

Have you checked for learned field extractions?

Look in /etc/apps/learned/local/

[jericksonpf]

not in there either

[kristian_kolb]

Some things to check.

In manager, make sure that you choose 'all apps' when you list the field extractions.

Have you looked in all props.conf files?

Have you looked in the correct place in the props.conf files? Fields can be extacted based on host or source as well as sourcetype.

Have you checked transforms.conf? Field extractions can happen there as well, if referenced from props.conf via the TRANSFORMS or REPORT attribute.

Have you tried to use btool, e.g. ./splunk cmd btool props list --debug

http://docs.splunk.com/Documentation/Splunk/5.0.4/Troubleshooting/CommandlinetoolsforusewithSupport#...

just what I came to think of off the top of my head...

/K

[kristian_kolb]

If you try;

splunk cmd btool props list EHR --debug

the results will be fewer, and only for matching props stanzas. Check with the other sourcetype as well, and the host / source if applicable.

I'm not sure if you can put field extraction configs in a user directory, at least I've never done it. But do check for props.conf files in etc/users and the directories in there. Not sure if btool looks in those places unless instructed to.

Oh, and make sure that UserID is not a field-alias for another field... but that is also in props.conf

[jericksonpf]

I did look in the transforms as well but could not find anything.

In the props.conf files i have been searching for the field name as it appears in the UI. It appears as : UserID
with a lower case a next to it instead of a #. Does the a symbol mean anything?

I ran the btool like this ./splunk cmd btool props list --debug | grep UserID but it came up with nothing

[jericksonpf]

the sourcetypes are EHR and EHRPDF-45
defined on the forwarder inputs.conf. They are web application logs. The original data source are the D drive on our web servers

[lguinn2]

Which sourcetypes? Also, what is in the original data source?