topic Re: Troubleshoot missing data in Getting Data In

Troubleshoot missing data

emsecrist — Fri, 29 Jan 2021 18:17:14 GMT

A new custom app and index was created and successfully deployed to 37 clients, as seen in the Fowarder Management interface in my Deployment Server. However, I do not see any data when searching in splunk.

Here is the stanza for the new index:

[sap] repFactor = auto homePath = volume:primary/sap/db coldPath = volume:cold/sap/colddb thawedPath = /opt/splunk/var/lib/splunk/cold1/sap/thaweddb tstatsHomePath = volume:primary/sap/datamodel_summary frozenTimePeriodInSecs = 7776000

Here is the inputs.conf for the new app:

[monitor:///hana/shared/*/XXX00/*/trace] sourcetype = sap-hana-trace index = sap

I have checked the Splunk UF logs and don't see any errors. Any help would be much appreciated.

Re: Troubleshoot missing data

isoutamo — Fri, 29 Jan 2021 19:27:33 GMT

Can you see internal logs from those UFs? And have you enabled UF’s restart after deployment? Probably splunk user has read access to those files as you haven’t any errors on splunkd.log.

r. Ismo

Re: Troubleshoot missing data

richgalloway — Fri, 29 Jan 2021 19:37:29 GMT

It's possible the timestamps of the events are being mis-interpreted and the events mis-filed. Try searching a wider time window.

index=sap sourcetype=sap-hana-trace earliest=-1y latest=+1y

Re: Troubleshoot missing data

emsecrist — Fri, 29 Jan 2021 20:16:43 GMT

Thanks, tried that command, still no data.

Re: Troubleshoot missing data

emsecrist — Fri, 29 Jan 2021 20:23:52 GMT

Yes I am able to see the internal logs from those UFs, and yes the UF restart is enabled after deployment. I even manually restarted one of the UFs. I also switched to the splunk user and verified read access to the directory and files.

Re: Troubleshoot missing data

richgalloway — Fri, 29 Jan 2021 20:41:54 GMT

If you're desperate try

index=* source="/hana/*"

Have you confirmed the forwarder has read access to the file path it's trying to monitor?

How many files are in that path? I'm wondering if it's more than the UF can handle.

Re: Troubleshoot missing data

emsecrist — Fri, 29 Jan 2021 20:47:57 GMT

Yes I am desperate. 😀

Tried that search, no data found. The UF does have access to the directory. The directory contains 2715 files.

Re: Troubleshoot missing data

isoutamo — Fri, 29 Jan 2021 22:42:25 GMT

What splunk list inputstatus shows on UF?

Probably you have already check these: https://wiki.splunk.com/Community:Troubleshooting_Monitor_Inputs

r. Ismo

Re: Troubleshoot missing data

scelikok — Sat, 30 Jan 2021 06:21:18 GMT

Hi @emsecrist,

Do you see an ERROR, WARN or started watching file log in below search?

index=_internal host=uf_hostname *hana*

Re: Troubleshoot missing data

emsecrist — Mon, 01 Feb 2021 18:41:14 GMT

No ERROR or WARN, but I do see this:

01-29-2021 09:36:05.565 -0700 INFO TailingProcessor - Adding watch on path: /hana/shared. • host = XXXXXXX • source = /opt/splunkforwarder/var/log/splunk/splunkd.log sourcetype = splunkd 01-29-2021 09:36:05.565 -0700 INFO TailingProcessor - Parsing configuration stanza: monitor:///hana/shared/*/HDB00/*/trace. • host = XXXXXXX • source = /opt/splunkforwarder/var/log/splunk/splunkd.log • sourcetype = splunkd

Re: Troubleshoot missing data

emsecrist — Mon, 01 Feb 2021 18:42:49 GMT

That is a helpful command, 'splunk list inputstatus', it provides a lot of information! I am doing some analysis of the output, I will let you know if I find anything interesting.

Re: Troubleshoot missing data

scelikok — Mon, 01 Feb 2021 18:57:34 GMT

Hi @emsecrist,

Can you please test by adding file extensions to monitor stanza like below?

[monitor:///hana/shared/*/XXX00/*/trace/*.log] sourcetype = sap-hana-trace index = sap

Re: Troubleshoot missing data

emsecrist — Mon, 01 Feb 2021 19:21:51 GMT

The files are *trc, and the sub-directory under shared always end in 1, so I updated the monitor line like so:

[monitor:///hana/shared/*1/XXX00/*/trace/*.trc]

I then ran 'splunk list inputstatus' from the UF, and it looks like splunk is seeing the correct log files. Here is a snippet from the output:

/hana/shared/SE1/XXX00/xxxxxxx/trace/system_availability_xxxxxxxxxxxx.trc file position = 39983 file size = 39983 parent = /hana/shared/*1/XXX00/*/trace/*.trc percent = 100.00 type = finished reading /hana/shared/SE1/HDB00/xxxxxxx/trace/webdispatcher_xxxxxxxxxx.30006.000.trc file position = 257800 file size = 257800 parent = /hana/shared/*1/XXX00/*/trace/*.trc percent = 100.00 type = open file

I still don't see any data in the 'sap' index though.

Re: Troubleshoot missing data

scelikok — Mon, 01 Feb 2021 19:39:39 GMT

Hi @emsecrist,

Splunk seems successfully hit the correct files. I believe these trace files are not plain text. That is why it cannot index. Maybe you can use DBConnect to query these trace info.

Re: Troubleshoot missing data

emsecrist — Mon, 01 Feb 2021 19:46:22 GMT

The data shows up now! I re-did the search with a longer time range, and it's there. Thank you for the help!

Re: Troubleshoot missing data

emsecrist — Mon, 01 Feb 2021 19:53:47 GMT

@scelikok One more question... If I wanted to monitor all the log files in the .../trace directory, how could I do that?

Re: Troubleshoot missing data

scelikok — Mon, 01 Feb 2021 20:12:28 GMT

Great! 😊

You can try below;

[monitor:///hana/shared/*/XXX00/*/trace/*] sourcetype = sap-hana-trace index = sap

Re: Troubleshoot missing data

Rajeev — Fri, 11 Jun 2021 07:34:44 GMT

Hi @emsecrist ,

Did you create the custom sourcetype "sap-hana-trace"? Would you mind sharing the config (from props.conf) for this custom sourcetype.

Re: Troubleshoot missing data

SamHTexas — Wed, 29 Sep 2021 19:59:44 GMT

Hello Bro., would you share how I would look for a list of "missing indexes" in Splunk Ent. please? The error I get says: Events not received from missing indexes". How would I fix "missing indexes issues" please? Thanks a million. teşekkürler.

Re: Troubleshoot missing data

scelikok — Sat, 16 Oct 2021 17:02:56 GMT

Hi @SamHTexas,

The solution is to create those missing indexes shown on messages on your indexers. I assume you are running clustered environment, you should do this on your cluster master and distribute it to your indexers.

If you want to check if you have an input that setup for non-exist index, you should check all your inputs.conf files for index values and compare them with your cluster indexes.conf files.