All Apps and Add-ons

NMON Performance Monitor for Unix and Linux Systems: TA_nmon app producing data on universal forwarder but not going to indexer

kmarx
Explorer

I have a 50G dev license sandbox where I've installed NMON on the indexer and TA_nmon on one of the universal forwarders (manually since my dev instance doesn't seem to allow a deployment server). But I never see data arrive at the indexer.

On the forwarder, I can see csv files cyclically come and go in

/opt/splunkforwarder/var/log/nmon/var/csv_repository/

But nothing ever shows up on the indexer. E.g., index=mon or index=*mon* show no results.

[Note that the above us under .../var/log/ on my install and not .../var/run/ per the trouble shooting article]

If I search on index=_internal host=myUFHost *nmon* I see lots of results saying things like:

WatchedFile - WatchedFile - Checksum for seekptr didn't match, will re-read entire file='/opt/splunkforwarder/var/log/nmon/var/csv_repository/dev-app01_57_VM.nmon.csv'.

and

WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/nmon/var/csv_repository/dev-app01_11_VM.nmon.csv'.

If I constrain the search for a given file=, I can see that at least some these messages repeat roughly hourly for a given file name. (I'm guessing the numbers are minute w/in the given hour?)

I did some searching on these messages and saw some suggestion that perhaps the UF tries to read the file before it's populated? Or perhaps it's getting deleted before processing completes?

With some help from folks on the Splunk Slack#getting-data-in channel I blithely tried index=_internal "drop" "index" and got a few hits like this on sourcetype=mongod:

2019-07-18T22:01:01.226Z I STORAGE  [conn967] dropCollection: s_nmon1Dpb033BBAauqdcA1GXmim53_kv_nmoyLxvM60i16Ei2OkLQ@wn5GLC.c (7bdb7e61-4fa5-48ff-bf30-2fe97841eaa6) - index namespace 's_nmon1Dpb033BBAauqdcA1GXmim53_kv_nmoyLxvM60i16Ei2OkLQ@wn5GLC.c.$_UserAndKeyUniqueIndex' would be too long after drop-pending rename. Dropping index immediately.

Any guidance would be greatly appreciated.

Platform:
- Splunk Enterprise 7.0.3
- Linux RHEL5 64bit (2.6.18-419.el5)

Places I've looked:
- https://answers.splunk.com/answers/400165/nmon-performance-monitor-for-unix-and-linux-system-5.html
- http://nmonsplunk.wikidot.com/documentation:userguide:troubleshoot:troubleguide
- https://answers.splunk.com/answers/126878/what-more-can-i-do-to-solve-file-too-small-to-check-seekcr...

Thanks!

0 Karma
1 Solution

guilmxm
SplunkTrust
SplunkTrust

Hi @kmarx

Sorry for the late reply.

Right, first the good troubleshooting link is the following:

http://nmon-for-splunk.readthedocs.io/en/latest/Userguide.html#troubleshooting-guide-from-a-to-z

It does not have anything to see with MongoDB at this stage, have you made sure that you created the nmon index in your standalone indexer instance ?
Because the logs you show from the forwarder looks ok, I'm not sure to see another possibility as you seem to have your forwarder forwarding the internal data to your indexer and the forwarder's nmon logs looks right.

Guilhem

View solution in original post

0 Karma

guilmxm
SplunkTrust
SplunkTrust

Hi @kmarx

Sorry for the late reply.

Right, first the good troubleshooting link is the following:

http://nmon-for-splunk.readthedocs.io/en/latest/Userguide.html#troubleshooting-guide-from-a-to-z

It does not have anything to see with MongoDB at this stage, have you made sure that you created the nmon index in your standalone indexer instance ?
Because the logs you show from the forwarder looks ok, I'm not sure to see another possibility as you seem to have your forwarder forwarding the internal data to your indexer and the forwarder's nmon logs looks right.

Guilhem

0 Karma

kmarx
Explorer

Hi @guilmxm and thanks for the reply. I didn't realize that I had to create the nmon index manually. I can do this but can you point me to doc on what settings I need to specify? E.g., events vs. metrics, anything else? I assume the App should be "NMON Performance by Octamis". Thank you!

0 Karma

guilmxm
SplunkTrust
SplunkTrust

No Pb
It’s actually in the doc:

https://nmon-for-splunk.readthedocs.io/en/latest/about.html#index-creation

The default index name we search is « nmon » and it’s an event index you create.

A better version of the app is available in Splunk Base and is called Metricator, better because it uses the metric store and type of indexes would be metrics.

Guilhem

0 Karma

kmarx
Explorer

Excellent. It does clearly state

An index called “nmon” must be created manually by Splunk administrators to use the default TA-nmon indexing parameters. (this can be tuned)

I know you didn't just add that now, but I was so sure I searched the doc for stuff like this. I'm seeing the indexed data now. Thanks very much!

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...