topic Re: How to check missing fields and not generating 100 percentage in Splunk Search

How to check missing fields and not generating 100 percentage

sasankganta — Mon, 04 Jan 2021 11:55:45 GMT

index="*" sourcetype="*" and I have field name with tag and it's generating 80% of events , how can I check why it's not generating 100% of events and where it's missing.

Also , Is there any regex / resi api / rex query to get what are all the fields generating 100% events , and if it's <100% whe

Re: How to check missing fields and not generating 100 percentage

richgalloway — Fri, 15 Jan 2021 15:05:32 GMT

It's normal for a field to be present in some events and not in others. The only fields we can be sure will have 100% coverage are _time, index, source, sourcetype, and host.

If you have a certain field that is supposed to be in all events, but if not then try this query to find the events that are missing it.

index=foo sourcetype=bar NOT field=*

The rex command extracts fields, but won't tell you which fields are in all events.

The regex command can filter out events that are missing certain fields. That's not much different from the query above, however.

| regex field!=".*"

I'm not aware of any REST command that returns field information.

If you're trying to test the CIM compliance of your data then try the CIM Validator app at https://splunkbase.splunk.com/app/2968/

Re: How to check missing fields and not generating 100 percentage

sasankganta — Fri, 15 Jan 2021 14:39:25 GMT

Thank you Mate for the documents. Yes for CIM validation, but don't have git in environment let me check possible ways. If you have any other suggestions will be grateful .

Re: How to check missing fields and not generating 100 percentage

richgalloway — Fri, 15 Jan 2021 15:17:03 GMT

You don't need git. Just download the app from github and install it on Splunk.

Re: How to check missing fields and not generating 100 percentage

sasankganta — Fri, 15 Jan 2021 15:27:03 GMT

Also one more, Index=A sourcetype=B and I can see under fields category filed "C" with count of 10k+ values ..

But if I search with Index=A sourcetype=B category=C , It is showing No results found tried in all the search modes didn't worked. source tcp:9997 . Can you please suggest what can be the issue.

Re: How to check missing fields and not generating 100 percentage

richgalloway — Fri, 15 Jan 2021 15:49:06 GMT

I don't have an answer. Sorry.