How does one get at fields in _internal
that are prefixed with an underscore, e.g. _tcp_KBps
? It seems that Splunk is masking these somehow, preventing them from being visible to stats, timechart, etc.
If you want to look at the internal metrics log (index=_internal source="*metrics.log*") you should see the attributes show up (like kbps, kb, eps) and you'll have access to report on them. Not sure what exactly you are looking to report on but there are some great examples from our wiki below from the Deployment monitor. You'll also find that the Splunk on Splunk App (S.O.S.) gives you dashboard views on many internal metrics to give you a deeper look at how Splunk is performing.
Use eval
to create "non-internal" fields out of them. These fields will be available to subsequent commands. Like this:
... | eval tcp_KBps=_tcp_KBps | ...
If you want to look at the internal metrics log (index=_internal source="*metrics.log*") you should see the attributes show up (like kbps, kb, eps) and you'll have access to report on them. Not sure what exactly you are looking to report on but there are some great examples from our wiki below from the Deployment monitor. You'll also find that the Splunk on Splunk App (S.O.S.) gives you dashboard views on many internal metrics to give you a deeper look at how Splunk is performing.