topic Re: Field extraction of a field that already exists in Splunk Search

How to do a field extraction of a field that already exists?

neerajs_81 — Tue, 19 Apr 2022 16:47:15 GMT

Gentlemen,
We are on Splunk Cloud.

In my raw events coming from AWS , splunk by default shows a field called "category" under "Interesting fields" . However, it's value ( as in it's extraction) isn't what we are expecting it to be. It only manages to extract a part of the complete string.

For example:
The raw events have category as follows (In JSON format)
"Policy:IAMUser/RootCredentialUsage" (without quotes)

But Splunk is instead showing the value of category as:  Policy

.Now, whats happening is if i use the IFX or rex command to create a field extraction keeping the same name for my field i.e. category and value: Policy:IAMUser/RootCredentialUsage , my newly extracted field keeps getting overwritten with the default old values again . I am assuming this is because the names of the fields are same ( category) , so splunk takes its own precedence.

IS this the case of Index time vs Search Time field extraction conflict ?

How to make Splunk use whatever value my field extraction ( as in rex or IFX) is extracting for category and at the same time also retain its name as is ? Dont want the category field to display its old indexed value.

Re: Field extraction of a field that already exists

PickleRick — Mon, 18 Apr 2022 19:58:24 GMT

The question is where does this field come from. You're saying it's auto-extracted but if the value in raw event is completely different then the field must be either a calculated field or provided as indexed field by the source (are you pushing the events with HEC?).

It's kinda strange and you should check all your ingestion process and source/sourcetype definitions.

But if you do a | rex command to extract your own value of category field, it should overwrite whatever is included in the event. Are you sure you're not mixing upper/lowercase in field names?

Re: Field extraction of a field that already exists

neerajs_81 — Tue, 19 Apr 2022 04:41:19 GMT

Thanks for responding. Yes, the "category" field is provided as an indexed field. The data is getting ingested via HEC ( Splunk TA for Amazon Kinesis Firehose). Does anything change if its coming via HEC ?
Positive there is no issue of upper case/lower case mismatch

What do you recommend then, if the indexed field is not getting overwritten by whatever value my rex / IFX is extracting for it ?

Re: Field extraction of a field that already exists

PickleRick — Tue, 19 Apr 2022 05:00:21 GMT

To be fully honest, I've never investigated what happens if you have the indexed field and try to extract it search-time as well. But as I said, any field should be overwritable with explicit field manipulation in search-time.

Anyway, is there no possibility to change the source behaviour? Because it's very strange that it defines a field which is already present (with another value) within the event itself.

Of course you can simply define an extraction to another field name (like eventcategory or something like that) but that's counterintuitive and can be confusing.

Re: Field extraction of a field that already exists

neerajs_81 — Tue, 19 Apr 2022 05:05:17 GMT

Thanks, yeah maybe a different field name is the way to go.