I have a 50G dev license sandbox where I've installed NMON on the indexer and TA_nmon on one of the universal forwarders (manually since my dev instance doesn't seem to allow a deployment server). But I never see data arrive at the indexer.
On the forwarder, I can see csv files cyclically come and go in
But nothing ever shows up on the indexer. E.g.,
index=*mon* show no results.
[Note that the above us under .../var/log/ on my install and not .../var/run/ per the trouble shooting article]
If I search on
index=_internal host=myUFHost *nmon* I see lots of results saying things like:
WatchedFile - WatchedFile - Checksum for seekptr didn't match, will re-read entire file='/opt/splunkforwarder/var/log/nmon/var/csv_repository/dev-app01_57_VM.nmon.csv'.
WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/nmon/var/csv_repository/dev-app01_11_VM.nmon.csv'.
If I constrain the search for a given
file=, I can see that at least some these messages repeat roughly hourly for a given file name. (I'm guessing the numbers are minute w/in the given hour?)
I did some searching on these messages and saw some suggestion that perhaps the UF tries to read the file before it's populated? Or perhaps it's getting deleted before processing completes?
With some help from folks on the Splunk Slack#getting-data-in channel I blithely tried
index=_internal "drop" "index" and got a few hits like this on sourcetype=mongod:
2019-07-18T22:01:01.226Z I STORAGE [conn967] dropCollection: s_nmon1Dpb033BBAauqdcA1GXmim53_kv_nmoyLxvM60i16Ei2OkLQ@wn5GLC.c (7bdb7e61-4fa5-48ff-bf30-2fe97841eaa6) - index namespace 's_nmon1Dpb033BBAauqdcA1GXmim53_kv_nmoyLxvM60i16Ei2OkLQ@wn5GLC.c.$_UserAndKeyUniqueIndex' would be too long after drop-pending rename. Dropping index immediately.
Any guidance would be greatly appreciated.
- Splunk Enterprise 7.0.3
- Linux RHEL5 64bit (2.6.18-419.el5)
Places I've looked:
Sorry for the late reply.
Right, first the good troubleshooting link is the following:
It does not have anything to see with MongoDB at this stage, have you made sure that you created the nmon index in your standalone indexer instance ?
Because the logs you show from the forwarder looks ok, I'm not sure to see another possibility as you seem to have your forwarder forwarding the internal data to your indexer and the forwarder's nmon logs looks right.
Hi @guilmxm and thanks for the reply. I didn't realize that I had to create the nmon index manually. I can do this but can you point me to doc on what settings I need to specify? E.g., events vs. metrics, anything else? I assume the App should be "NMON Performance by Octamis". Thank you!
It’s actually in the doc:
The default index name we search is « nmon » and it’s an event index you create.
A better version of the app is available in Splunk Base and is called Metricator, better because it uses the metric store and type of indexes would be metrics.
Excellent. It does clearly state
An index called “nmon” must be created manually by Splunk administrators to use the default TA-nmon indexing parameters. (this can be tuned)
I know you didn't just add that now, but I was so sure I searched the doc for stuff like this. I'm seeing the indexed data now. Thanks very much!