Easily find the conf of an extracted field

[verbal_666]

A simple, but not trivial, question.
When searching a log file, we sometimes find fields previously extracted "on-the-fly" from the search SH, in various ways (Regex, Field_Delimiter), or directly from the Indexer, again in different ways.

NOW, the question: it often happens that we spend a lot of time searching for the exact config from which the extraction comes, sifting through the various SH and Indexer configs... eventually, between one thing and another, we get there. But, as mentioned, it often wastes a lot of time... just an example: i have a log where, in many years, many users put an hand, any user inserted his field extraction for a single field, creating something like 20/30 conf for a single log and for 20/30 fields... bothersome when debugging and seeking a single field (as said with the search form you get it quite soon, but...)!!! 🙄

Is there a quick way to understand WHERE a particular field extracted from the log COMES FROM? A "detailed-field-inspector" that tells us exactly: "this field is extracted from the SH, Regex, 'Sourcetype: my_extract_conf'"?

Thanks.

[tscroggins]

Hi @verbal_666,

If you're comfortable with the job inspector and reviewing search logs, you can learn a lot from the debug output of various components: *LookupDriver, CalcFieldProcessor, FieldAliaser, SearchOperator:*, etc.

index=foo bar baz qux
| noop log_debug=* remote_log_fetch=*
| extract reload=true

The output won't always show you where a configuration lives, but it will show you how a field was derived. Combined with knowledge of search-time operations and your own deployment architecture, you can source the configuration fairly quickly.

[verbal_666]

Very interesting 👍👍👍
Thanks a lot 👏👏

[livehybrid]

Hi @verbal_666 

I see that the general consensus here is that its not possible, but you might find that you can achieve what you need using btool? I often use btool via  Admins Little Helper app's custom btool commad, such as:

|btool props list | search "myCustomField"

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[verbal_666]

Really useful 👍👍👍

Many thanks 👏👏👍

[verbal_666 · Accepted solution]

for my knowledge, there isn't!

Short answer is - no.

I think the issue is totally resolved 😂

Thanks all 😎

[gcusello]

Hi @verbal_666 ,

as I said: add this proposal to ideas.splunk.com; I'll vote for it!

Ciao.

Giuseppe

[PickleRick]

Short answer is - no.

Long answer is - there is no technical possibility to trace back statically that information. While you might be able to pinpoint "after a lot of time searching for the exact config" where the field came from, it is not possible without the results. Simply because some fields are not statically defined, but are dynamic.

The easiest example of this would be fields extracted by automatic key-value parsing. More interesting case would be dynamic field creation with brackets syntax (eval {something}="here I am!").

So if you wanted to have this information you'd have to:

1) Have a complete audit trace of every single operation in the search pipeline

2) Drag this monstrosity along with the results from the indexers to the searchheads... (not to mention cases like federated search).

So as usual - the question is simple, the answer isn't.

[gcusello]

Hi @verbal_666 ,

for my knowledge, there isn't!

there's only the information in which app a field configuration (extraction, tramsformation, calculation, etc...) is defined, but there isn't any information on index time extractions.

In theory, you should be much structured in your activity to know and document where a configuration is done, nut rarely I found this this , especially if more people work on the same project.

Add this proposal to ideas.splunk.com; I'll vote for it!

Ciao.

Giuseppe