Unexpected results with field values - Splunk Enterprise

[akarivaratharaj]

With some of the events, we are facing the unexpected format of the query results. Actually in the raw event there is no issue at all, and each field is showing their own values. But when it is queried and displayed in the statistics section as results, the values of few fields are displaying incorrectly.

Usually the search results show key-values. But with some events, the search results are showing as "fieldname1=fieldname1=value" and in some cases "fieldname1=fieldname3=value". 

Example1: Request_id=Request_id=12345

(Expected to be -> "Request_id=12345")

Example2: Parent_id=message_id=456

(Expected to be -> "Parent_id=321")

Example3: Parent_id=category=unknown

(Expected to be -> "Parent_id=321")

Is this related with parser or something else? We are unable to find what could be the issue lying over here.

Could anyone please help us on fixing this issue at the earliest?

[akarivaratharaj]

This issue is resolved after making few changes to props.conf where the field extraction is set.

[ITWhisperer]

It looks to like you either have a problem with your data (raw events), your ingest config e.g. transforms.conf or your search query. Unfortunately, since you have shared none of these, it is rather difficult to offer anything more constructive.

[akarivaratharaj]

If I run a search query, there is no issue with raw events. From the Events tab, everything looks in perfect format and can't say that there is a Data quality issue in the events.

Only when this is visualised from statistics tab I could see this. Also this is happening only with some events in the results set. I have attached the screenshot of the normal results and the results with Data Quality issue.

Expected results with Request Id and other fields.

But what it is displaying (Refer the highlighted rows)

Here is the event of one of the request ids where the key value pair is as expected format

[ITWhisperer]

What is the search?

[ITWhisperer]

How are the fields extracted?

[akarivaratharaj]

I am using just the table command

index=main host=* sourcetype=* source=* | table _time, Request_id, Future_id

[ITWhisperer]

So it looks like it is to do with how the fields are extracted. Please can you share these details?

[ITWhisperer]

Also you shared (a picture of) an event which works, but not one which doesn't. Please can you share the raw text of a "failing" event in a code block (rather than a picture) - you can obfuscate any sensitive details as appropriate.

[akarivaratharaj]

Actually I have shared picture of the raw event of the failed ones only (just masked the confidential fields). They look similar to the other events which work.

[ITWhisperer]

So it looks like it is to do with how the fields are extracted. Please can you share these details?

[akarivaratharaj]

I have observed one more thing with these failed events. In the event section, usually at the end of each event, the default fields like host, sourcetype, etc., will be appended and displayed.

Similarly, in addition to those default fields, I could see the Request_ID field is also displayed in that section after each event. In that place I could see the format of Request_ID is in unexpected form.

Please check the below screenshot (After the field CT=1, the section of default fields is shown)

[ITWhisperer]

This is showing that the fields have been extracted incorrectly.

Yet again I ask if you could please share your configurations which are being used to extract the fields for this sourcetype - this is likely to be where your problem lies, so if you want a resolution, you are going to have to give us more information.