Gentlemen,
We are on Splunk Cloud.
In my raw events coming from AWS , splunk by default shows a field called "category" under "Interesting fields" . However, it's value ( as in it's extraction) isn't what we are expecting it to be. It only manages to extract a part of the complete string.
For example:
The raw events have category as follows (In JSON format)
"Policy:IAMUser/RootCredentialUsage" (without quotes)
But Splunk is instead showing the value of category as: Policy
.Now, whats happening is if i use the IFX or rex command to create a field extraction keeping the same name for my field i.e. category and value: Policy:IAMUser/RootCredentialUsage , my newly extracted field keeps getting overwritten with the default old values again . I am assuming this is because the names of the fields are same ( category) , so splunk takes its own precedence.
IS this the case of Index time vs Search Time field extraction conflict ?
How to make Splunk use whatever value my field extraction ( as in rex or IFX) is extracting for category and at the same time also retain its name as is ? Dont want the category field to display its old indexed value.
The question is where does this field come from. You're saying it's auto-extracted but if the value in raw event is completely different then the field must be either a calculated field or provided as indexed field by the source (are you pushing the events with HEC?).
It's kinda strange and you should check all your ingestion process and source/sourcetype definitions.
But if you do a | rex command to extract your own value of category field, it should overwrite whatever is included in the event. Are you sure you're not mixing upper/lowercase in field names?
Thanks for responding. Yes, the "category" field is provided as an indexed field. The data is getting ingested via HEC ( Splunk TA for Amazon Kinesis Firehose). Does anything change if its coming via HEC ?
Positive there is no issue of upper case/lower case mismatch
What do you recommend then, if the indexed field is not getting overwritten by whatever value my rex / IFX is extracting for it ?
To be fully honest, I've never investigated what happens if you have the indexed field and try to extract it search-time as well. But as I said, any field should be overwritable with explicit field manipulation in search-time.
Anyway, is there no possibility to change the source behaviour? Because it's very strange that it defines a field which is already present (with another value) within the event itself.
Of course you can simply define an extraction to another field name (like eventcategory or something like that) but that's counterintuitive and can be confusing.
Thanks, yeah maybe a different field name is the way to go.