Why is my license usage search by app not showing information for all apps in my environment?

[sunnyparmar]

Hi,

I have a search mentioned below to see license usage per app, but the issue I am facing is, if I run the search without this portion ( | chart sum(GB) by app_name) at the end, then it gives the license usage for all apps with their indexes and sourcetypes respectively. However, if I exclude it from the end, then it is not showing license usage for all the apps. Kindly suggest for this.

index=_internal source=*license_usage.log type="Usage" | eval indexname = if(len(idx)=0 OR isnull(idx),"(UNKNOWN)",idx)  | eval sourcetypename = st  | bin _time span=1d  | stats sum(b) as b by _time, pool, indexname, sourcetypename | eval GB=round(b/1024/1024/1024, 3)| fields _time, indexname, sourcetypename, GB |  join sourcetypename [       | rest /services/saved/sourcetypes       | fields title, "eai:acl.app"       | rename title AS sourcetypename, "eai:acl.app" AS app_name  ] | chart sum(GB) by app_name

Thanks

[renjith_nair]

Not sure exactly but the same search works for me 🙂

index=_internal source="*license_usage.log" type="Usage" 
| eval indexname = if(len(idx)=0 OR isnull(idx),"(UNKNOWN)",idx) | eval sourcetypename = st | bin _time span=1d 
| stats sum(b) as b by _time, pool, indexname, sourcetypename | eval GB=round(b/1024/1024/1024, 3)
| fields _time, indexname, sourcetypename, GB 
| join sourcetypename [ | rest /services/saved/sourcetypes | fields title, "eai:acl.app" | rename title AS sourcetypename, "eai:acl.app" AS app_name ]

[sunnyparmar]

thanks for the correction. By executing that query i am getting data in "Not Found".

Index Sourcetype
qv-perfmon Perfmon:CPUTime

qv-perfmon Perfmon:FreeDiskSpace

qv-perfmon Perfmon:Memory

qv-winevents WinEventLog:Application

qv-winevents WinEventLog:Security

qv-winevents WinEventLog:System

But now the issue here is it is very common sourcetype that we used in splunk for making alerts for host health checks and make entries in inputs.conf so in this case could you please tell me how to overcome this issue?

Thanks

[renjith_nair]

Your alerts/searches are not aaffected by this. The above sourcetypes might not be part of any app and hence app_name is not displayed.

Again to find out the missing apps , let's do step by step

1. Take one of the apps from the missing list (let's say app1)
2. Find out its sourcetypes | rest /services/saved/sourcetypes | fields title, "eai:acl.app" | rename title AS sourcetypename, "eai:acl.app" AS app_name|search app_name="app1"
3. Check those sourcetypes are part of your license search index=_internal source="*license_usage.log" type="Usage" st="sourcetypes from stpe2"
4. If you don't get any results for step3 , the app1 is not sending any data and hence its missing since you have a join to map the sourcetypes. So it results only those apps which has a matching sourcetype in your "license search"
5. In case you find it out, try changing the time range of your initial search to something bigger ,let's say last 30 days

[sunnyparmar]

Well sorry once again as i am executing wrong previously.. so now what I comes up with final result is after running 2nd point query is -
sourcetypename app_name
Alusta_Validation Alusta_Tenant_Validation
and when i am providing the sourcetype in 3rd point query so it is giving "no result found". so now could you please tell me that how to merge Alusta_validation sourcetype with join.

Thanks

[renjith_nair]

Can you try this for the time range you are using in your license search

index=* sourcetype="Alusta_Validation"|stats count

If you are not getting any result for this, there is no data indexing for this sourcetype. If there is no data indexing for the sourcetype, it does not make sense to include it as part of license usage

[sunnyparmar]

it is showing the last logs of September 2015. After that there is no logs detail.

[renjith_nair]

Hope now you are clear about the issue. Your time range might not reach Sep 2015 and hence the corresponding app is not listed. This is correct and not an issue. If you want this app to be listed, investigate why this app/related inputs are not sending any data and resolve the issue.

Hope this clarifies

[sunnyparmar]

thanks a lot.. yes it clarifies now but this rule is implemented in case of 1 app. but in other two apps, i am getting data for those for current date means today but why they are not enlisting in result. Could you please clarify this if i am running your giving query (| rest /services/saved/sourcetypes | fields title, "eai:acl.app" | rename title AS sourcetypename, "eai:acl.app" AS app_name) both of them not enlisted in the result. so do you have any idea what's the issue here?

Thanks

[renjith_nair]

It's just a part of your original search. For the search | rest /services/saved/sourcetypes | fields title, "eai:acl.app" | rename title AS sourcetypename, "eai:acl.app" AS app_name|search app_name="app2" where app2 is your second missing app and if it's not have any result, then none of it's souretypes might be sending dat. But you have mentioned above that you are getting the log files for the apps today. How did you verify this ? I'm sorry but I don't have any idea now

[sunnyparmar]

i have made dashboard for those apps and getting data on those app panels on daily basis. One more thing i want to ask here if you can suggest me please that one of my app is License Usage and all those panels in that app using index=internal and in my main query also i am using _internal index for enlisting all the data so could you please suggest me here that why my License usage app is not coming in final result and it is getting data also on daily basis.

Thanks

[sunnyparmar]

hey.. do you have any idea that in my current query if i have to add metrics.log as a source so how can i do this. If this will happen it will resolve one of my app problem. currently i am using only license_usage log as a source so how to add one more source in this query. Query is same -

index=_internal source=*license_usage.log OR source=/opt/splunk/var/log/splunk/metrics.log type="Usage" | eval indexname = if(len(idx)=0 OR isnull(idx),"(UNKNOWN)",idx)  | eval sourcetypename = st  | bin _time span=1d  | stats sum(b) as b by _time, pool, indexname, sourcetypename | eval GB=round(b/1024/1024/1024, 3)| fields _time, indexname, sourcetypename, GB |  join sourcetypename [       | rest /services/saved/sourcetypes       | fields title, "eai:acl.app"       | rename title AS sourcetypename, "eai:acl.app" AS app_name  ] | chart sum(GB) by app_name

Thanks in Advance

[renjith_nair]

Did you manage to make it working?

[sunnyparmar]

hey, on executing your both queries (2 & 3) it is giving me "no result found". Thanks

[sunnyparmar]

have you executed the query with adding parameter (| chart sum(GB) by app_name) in the last. Please let me know if it will show you all your apps in the result. Thanks

[renjith_nair]

Yes I have only 2 apps in my test and they are showing like below

app_name                                                   sum(GB)
search                                                  0.000
splunk_app_windows_infrastructure   0.001

Final search I ran :

index=_internal source="*license_usage.log" type="Usage" 
 | eval indexname = if(len(idx)=0 OR isnull(idx),"(UNKNOWN)",idx) | eval sourcetypename = st | bin _time span=1d 
 | stats sum(b) as b by _time, pool, indexname, sourcetypename | eval GB=round(b/1024/1024/1024, 3)
 | fields _time, indexname, sourcetypename, GB 
 | join sourcetypename [ | rest /services/saved/sourcetypes | fields title, "eai:acl.app" | rename title AS sourcetypename, "eai:acl.app" AS app_name ]
 | chart sum(GB) by app_name

Are you getting some apps or not at all ?

[sunnyparmar]

i am getting some app but not all.. what i want is to enlisted all the apps in the final result. Just don't understand why it behaves weird?

[renjith_nair]

can you run below search and see if all these apps are showing in the original search ?

index=_internal source="*license_usage.log" type="Usage" 
  | eval indexname = if(len(idx)=0 OR isnull(idx),"(UNKNOWN)",idx) | eval sourcetypename = st | bin _time span=1d 
  | stats sum(b) as b by _time, pool, indexname, sourcetypename | eval GB=round(b/1024/1024/1024, 3)
  | fields _time, indexname, sourcetypename, GB 
  | join sourcetypename [ | rest /services/saved/sourcetypes | fields title, "eai:acl.app" | rename title AS sourcetypename, "eai:acl.app" AS app_name ]|table app_name|dedup app_name

OR

    index=_internal source="*license_usage.log" type="Usage" 
      | eval indexname = if(len(idx)=0 OR isnull(idx),"(UNKNOWN)",idx) | eval sourcetypename = st | bin _time span=1d 
      | stats sum(b) as b by _time, pool, indexname, sourcetypename | eval GB=round(b/1024/1024/1024, 3)
      | fields _time, indexname, sourcetypename, GB 
      | join sourcetypename [ | rest /services/saved/sourcetypes | fields title, "eai:acl.app" | rename title AS sourcetypename, "eai:acl.app" AS app_name ]|stats count by app_name

And also check the next pages if there are (its obvious but just in case)

[sunnyparmar]

I have executed both the queries and still 3 apps are missing in result in both queries..

[renjith_nair]

Are those three apps coming as part of original detailed search ? Also those three apps have common sourcetypename to result in join ?

[sunnyparmar]

Q - Are those three apps coming as part of original detailed search?
A - Yes..

Q - Also those three apps have common sourcetypename to result in join ?
A- Sorry, Didn't get you.