topic Re: Is the "type" field removed from Splunk metrics in 6.6? in Splunk Search

Is the "type" field removed from Splunk metrics in 6.6?

EricLloyd79 — Tue, 22 May 2018 15:58:54 GMT

So we have this query:

index=_internal type=Usage  st!=splunk_metrics earliest=-1d@d latest=-0d@d  | bucket _time span=1d | stats sum(eval(b/1024/1024/1024)) as GB by _time

Its been running on Splunk for years for us, producing some info about how much is being indexed per day .. we upgraded to Splunk 6.6 and it seems like it doesn't work anymore.
I don't see the field "type" anymore
Does anyone know if they changed this in this new version?

Re: Is the "type" field removed from Splunk metrics in 6.6?

PowerPacked — Tue, 29 Sep 2020 19:36:20 GMT

Hi @EricLloyd79

Can you give this search a try

index=_internal source=*license_usage.log type=Usage earliest=-1d@d latest=-0d@d | bucket _time span=1d | stats sum(eval(b/1024/1024/1024)) as GB by _time

Thanks

Re: Is the "type" field removed from Splunk metrics in 6.6?

somesoni2 — Tue, 22 May 2018 16:28:08 GMT

Before Splunk 6.5.x, Splunk used to report license data in a single log file license_usage.log. It used to differentiate frequent license usage vs daily rollover summary via field type that you used in the search above. Starting 6.5.x, the license rollover summary logs have been moved a dedicated log file called license_usage_summary.log (so all logs with type=RolloverSummary), thus the field type is removed. See below links for brief details on both the files (and other internal log files) in Splunk.

https://docs.splunk.com/Documentation/Splunk/6.6.0/Troubleshooting/WhatSplunklogsaboutitself#Internal_logs

Re: Is the "type" field removed from Splunk metrics in 6.6?

EricLloyd79 — Tue, 29 Sep 2020 19:39:58 GMT

I tried that search and got no results.

This search works to find specifically by host:
index="_internal" source="*metrics.log" group="per_host_thruput" | chart sum(kb) by series | sort - sum(kb)

Now, I was able to get results when I took type=Usage out:
index=_internal source=*license_usage.log earliest=-1d@d latest=-0d@d | bucket _time span=1d | stats sum(eval(b/1024/1024/1024)) as GB by _time

But I am beginning to suspect it has to do with, for some reason, we are unable to access license_usage.log on our licensing server.
We get this message:

LicenseUsage - type=Message - License usage logging not available for slave licensing instances, please see license_usage.log on license master=https://10.10.x.x:8089 for usage breakdown

I can see the license_usage.log file in our licensing server via CLI but when I run this query it can't seem to find it. We recently upgraded to 6.6 but I doubt that would have anything to do with it.

Re: Is the "type" field removed from Splunk metrics in 6.6?

EricLloyd79 — Tue, 29 Sep 2020 19:40:00 GMT

This is interesting. When I go on our licensing server and look at license_usage.log, I still see a Type=Usage being logged as of a few mins ago.

We are currently experiencing some unusual SSL error connecting to our licensing server when we run our script so I suspect that may be part of the issue that our original query isn't working:
index=_internal type=Usage st!=splunk_metrics earliest=-1d@d latest=-0d@d | bucket _time span=1d | stats sum(eval(b/1024/1024/1024)) as GB by _time

It is looking for a log file with type=Usage which only exists in the license_usage.log on the licensing manager which cannot be accessed. When I change it to type=* ( and remove st!=splunk metrics which seems like an artifact), I get these types:

Message - License usage logging not available for slave licensing instances
RolloverSummary
SlaveWarnSummary

This seems to correlate with what you are saying (sort of) and also is retrieving license_usage.log files from slave licensing instances which do not have the type=Usage field (hence why the original query got no results).

We will work on getting the SSL error resolved then go from there. Thanks for the info.

Re: Is the "type" field removed from Splunk metrics in 6.6?

EricLloyd79 — Tue, 22 May 2018 19:19:56 GMT

The problem was that for some reason when we upgrades, the inputs.conf changed the hostname of our licensing server (very odd) so once we fixed that it all worked correctly.