Troubleshoot missing data

[emsecrist]

A new custom app and index was created and successfully deployed to 37 clients, as seen in the Fowarder Management interface in my Deployment Server. However, I do not see any data when searching in splunk. 

Here is the stanza for the new index:

[sap]
repFactor  = auto
homePath   = volume:primary/sap/db
coldPath   = volume:cold/sap/colddb
thawedPath = /opt/splunk/var/lib/splunk/cold1/sap/thaweddb
tstatsHomePath = volume:primary/sap/datamodel_summary
frozenTimePeriodInSecs = 7776000

Here is the inputs.conf for the new app:

[monitor:///hana/shared/*/XXX00/*/trace]
sourcetype = sap-hana-trace
index = sap

I have checked the Splunk UF logs and don't see any errors. Any help would be much appreciated.

[Rajeev]

Hi @emsecrist ,

Did you create the custom sourcetype "sap-hana-trace"? Would you mind sharing the config (from props.conf) for this custom sourcetype.

[scelikok]

Hi @emsecrist,

Do you see an ERROR, WARN or started watching file log in below search?

index=_internal host=uf_hostname *hana*

[emsecrist]

No ERROR or WARN, but I do see this:

01-29-2021 09:36:05.565 -0700 INFO TailingProcessor - Adding watch on path: /hana/shared.
• host = XXXXXXX
• source = /opt/splunkforwarder/var/log/splunk/splunkd.log
sourcetype = splunkd

01-29-2021 09:36:05.565 -0700 INFO TailingProcessor - Parsing configuration stanza: monitor:///hana/shared/*/HDB00/*/trace.
• host = XXXXXXX
• source = /opt/splunkforwarder/var/log/splunk/splunkd.log
• sourcetype = splunkd

[scelikok · Accepted solution]

Hi @emsecrist,

Can you please test by adding file extensions to monitor stanza like below?

[monitor:///hana/shared/*/XXX00/*/trace/*.log]
sourcetype = sap-hana-trace
index = sap

[SamHTexas]

Hello Bro.,  would you share how I would look for a list of "missing indexes" in Splunk Ent.  please? The error I get says: Events not received from missing indexes". How would I fix "missing indexes issues" please? Thanks a million. teşekkürler.

[scelikok]

Hi @SamHTexas,

The solution is to create those missing indexes shown on messages on your indexers. I assume you are running clustered environment, you should do this on your cluster master and distribute it to your indexers.

If you want to check if you have an input that setup for non-exist index, you should check all your inputs.conf files for index values and compare them with your cluster indexes.conf files.

[emsecrist]

The files are *trc, and the sub-directory under shared always end in 1, so I updated the monitor line like so:

[monitor:///hana/shared/*1/XXX00/*/trace/*.trc]

 I then ran 'splunk list inputstatus' from the UF, and it looks like splunk is seeing the correct log files. Here is a snippet from the output:

/hana/shared/SE1/XXX00/xxxxxxx/trace/system_availability_xxxxxxxxxxxx.trc
                file position = 39983
                file size = 39983
                parent = /hana/shared/*1/XXX00/*/trace/*.trc
                percent = 100.00
                type = finished reading

        /hana/shared/SE1/HDB00/xxxxxxx/trace/webdispatcher_xxxxxxxxxx.30006.000.trc
                file position = 257800
                file size = 257800
                parent = /hana/shared/*1/XXX00/*/trace/*.trc
                percent = 100.00
                type = open file

 I still don't see any data in the 'sap' index though.

[scelikok]

Hi @emsecrist,

Splunk seems successfully hit the correct files. I believe these trace files are not plain text. That is why it cannot index. Maybe you can use DBConnect to query these trace info.

[emsecrist]

@scelikok One more question... If I wanted to monitor all the log files in the .../trace directory, how could I do that?

[scelikok]

Great! 😊 

You can try below;

[monitor:///hana/shared/*/XXX00/*/trace/*]
sourcetype = sap-hana-trace
index = sap

[emsecrist]

The data shows up now! I re-did the search with a longer time range, and it's there. Thank you for the help!

[richgalloway]

It's possible the timestamps of the events are being mis-interpreted and the events mis-filed.  Try searching a wider time window.

index=sap sourcetype=sap-hana-trace earliest=-1y latest=+1y

[emsecrist]

Thanks, tried that command, still no data.

[richgalloway]

If you're desperate try

index=* source="/hana/*"

Have you confirmed the forwarder has read access to the file path it's trying to monitor?  

How many files are in that path?  I'm wondering if it's more than the UF can handle.

[emsecrist]

Yes I am desperate. 😀

Tried that search, no data found. The UF does have access to the directory. The directory contains 2715 files.

[isoutamo]

Can you see internal logs from those UFs? And have you enabled UF’s restart after deployment? Probably splunk user has read access to those files as you haven’t any errors on splunkd.log.

r. Ismo

[emsecrist]

Yes I am able to see the internal logs from those UFs, and yes the UF restart is enabled after deployment. I even manually restarted one of the UFs. I also switched to the splunk user and verified read access to the directory and files.

[isoutamo]

What splunk list inputstatus shows on UF?

Probably you have already check these: https://wiki.splunk.com/Community:Troubleshooting_Monitor_Inputs

r. Ismo

[emsecrist]

That is a helpful command, 'splunk list inputstatus', it provides a lot of information! I am doing some analysis of the output, I will let you know if I find anything interesting.