Getting Data In

Troubleshoot missing data

emsecrist
Explorer

A new custom app and index was created and successfully deployed to 37 clients, as seen in the Fowarder Management interface in my Deployment Server. However, I do not see any data when searching in splunk. 

Here is the stanza for the new index:

[sap]
repFactor  = auto
homePath   = volume:primary/sap/db
coldPath   = volume:cold/sap/colddb
thawedPath = /opt/splunk/var/lib/splunk/cold1/sap/thaweddb
tstatsHomePath = volume:primary/sap/datamodel_summary
frozenTimePeriodInSecs = 7776000

Here is the inputs.conf for the new app:

[monitor:///hana/shared/*/XXX00/*/trace]
sourcetype = sap-hana-trace
index = sap

I have checked the Splunk UF logs and don't see any errors. Any help would be much appreciated.

Labels (4)
0 Karma
1 Solution

scelikok
SplunkTrust
SplunkTrust

Hi @emsecrist,

Can you please test by adding file extensions to monitor stanza like below?

[monitor:///hana/shared/*/XXX00/*/trace/*.log]
sourcetype = sap-hana-trace
index = sap

 

If this reply helps you an upvote is appreciated.

View solution in original post

Rajeev
Loves-to-Learn Lots

Hi @emsecrist ,

Did you create the custom sourcetype "sap-hana-trace"? Would you mind sharing the config (from props.conf) for this custom sourcetype.

0 Karma

scelikok
SplunkTrust
SplunkTrust

Hi @emsecrist,

Do you see an ERROR, WARN or started watching file log in below search?

index=_internal host=uf_hostname *hana*

 

If this reply helps you an upvote is appreciated.

emsecrist
Explorer

No ERROR or WARN, but I do see this:

01-29-2021 09:36:05.565 -0700 INFO TailingProcessor - Adding watch on path: /hana/shared.
• host = XXXXXXX
• source = /opt/splunkforwarder/var/log/splunk/splunkd.log
sourcetype = splunkd

01-29-2021 09:36:05.565 -0700 INFO TailingProcessor - Parsing configuration stanza: monitor:///hana/shared/*/HDB00/*/trace.
• host = XXXXXXX
• source = /opt/splunkforwarder/var/log/splunk/splunkd.log
• sourcetype = splunkd
0 Karma

scelikok
SplunkTrust
SplunkTrust

Hi @emsecrist,

Can you please test by adding file extensions to monitor stanza like below?

[monitor:///hana/shared/*/XXX00/*/trace/*.log]
sourcetype = sap-hana-trace
index = sap

 

If this reply helps you an upvote is appreciated.

SamHTexas
Builder

Hello Bro.,  would you share how I would look for a list of "missing indexes" in Splunk Ent.  please? The error I get says: Events not received from missing indexes". How would I fix "missing indexes issues" please? Thanks a million. teşekkürler.

Tags (1)
0 Karma

scelikok
SplunkTrust
SplunkTrust

Hi @SamHTexas,

The solution is to create those missing indexes shown on messages on your indexers. I assume you are running clustered environment, you should do this on your cluster master and distribute it to your indexers.

If you want to check if you have an input that setup for non-exist index, you should check all your inputs.conf files for index values and compare them with your cluster indexes.conf files.

If this reply helps you an upvote is appreciated.
0 Karma

emsecrist
Explorer

The files are *trc, and the sub-directory under shared always end in 1, so I updated the monitor line like so:

[monitor:///hana/shared/*1/XXX00/*/trace/*.trc]

 I then ran 'splunk list inputstatus' from the UF, and it looks like splunk is seeing the correct log files. Here is a snippet from the output:

/hana/shared/SE1/XXX00/xxxxxxx/trace/system_availability_xxxxxxxxxxxx.trc
                file position = 39983
                file size = 39983
                parent = /hana/shared/*1/XXX00/*/trace/*.trc
                percent = 100.00
                type = finished reading

        /hana/shared/SE1/HDB00/xxxxxxx/trace/webdispatcher_xxxxxxxxxx.30006.000.trc
                file position = 257800
                file size = 257800
                parent = /hana/shared/*1/XXX00/*/trace/*.trc
                percent = 100.00
                type = open file

 I still don't see any data in the 'sap' index though.

0 Karma

scelikok
SplunkTrust
SplunkTrust

Hi @emsecrist,

Splunk seems successfully hit the correct files. I believe these trace files are not plain text. That is why it cannot index. Maybe you can use DBConnect to query these trace info.

If this reply helps you an upvote is appreciated.

emsecrist
Explorer

@scelikok One more question... If I wanted to monitor all the log files in the .../trace directory, how could I do that?

0 Karma

scelikok
SplunkTrust
SplunkTrust

Great! 😊 

You can try below;

[monitor:///hana/shared/*/XXX00/*/trace/*]
sourcetype = sap-hana-trace
index = sap
If this reply helps you an upvote is appreciated.

emsecrist
Explorer

The data shows up now! I re-did the search with a longer time range, and it's there. Thank you for the help!

0 Karma

richgalloway
SplunkTrust
SplunkTrust

It's possible the timestamps of the events are being mis-interpreted and the events mis-filed.  Try searching a wider time window.

index=sap sourcetype=sap-hana-trace earliest=-1y latest=+1y
---
If this reply helps you, Karma would be appreciated.

emsecrist
Explorer

Thanks, tried that command, still no data.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

If you're desperate try

index=* source="/hana/*"

Have you confirmed the forwarder has read access to the file path it's trying to monitor?  

How many files are in that path?  I'm wondering if it's more than the UF can handle.

 

---
If this reply helps you, Karma would be appreciated.

emsecrist
Explorer

Yes I am desperate. 😀

Tried that search, no data found. The UF does have access to the directory. The directory contains 2715 files.

 

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Can you see internal logs from those UFs? And have you enabled UF’s restart after deployment? Probably splunk user has read access to those files as you haven’t any errors on splunkd.log.

r. Ismo

emsecrist
Explorer

Yes I am able to see the internal logs from those UFs, and yes the UF restart is enabled after deployment. I even manually restarted one of the UFs. I also switched to the splunk user and verified read access to the directory and files.

0 Karma

isoutamo
SplunkTrust
SplunkTrust

What splunk list inputstatus shows on UF?

Probably you have already check these: https://wiki.splunk.com/Community:Troubleshooting_Monitor_Inputs

r. Ismo

emsecrist
Explorer

That is a helpful command, 'splunk list inputstatus', it provides a lot of information! I am doing some analysis of the output, I will let you know if I find anything interesting.

0 Karma
Get Updates on the Splunk Community!

How I Instrumented a Rust Application Without Knowing Rust

As a technical writer, I often have to edit or create code snippets for Splunk's distributions of ...

Splunk Community Platform Survey

Hey Splunk Community, Starting today, the community platform may prompt you to participate in a survey. The ...

Observability Highlights | November 2022 Newsletter

 November 2022Observability CloudEnd Of Support Extension for SignalFx Smart AgentSplunk is extending the End ...