How find amount not ingested?

[sverdhan]

i have used the below query to get a list of 25 sourcetypes who are not reporting for the last 30 days ...but i need to know the volume of data ingested by them...kindly suggest any ideas or any alternative methods:

| metadata type=sourcetypes | eval diff=now()-lastTime | where diff > 3600*24*30 | convert ctime(lastTime) | convert ctime(firstTime) | convert ctime(recentTime) | sort -diff

[gcusello]

Hi @sverdhan ,

sorry but your requirement isn't so clear:

if a sourcetype didn't reported in the last 30 days, how can you calculate their volume? it's always 0 in the last 30 days.

Maybe you want the logs in the last 6 months, calculating their total volume highlighting if they aren't sending logs from 30 days.

in this case, you can apply a solution like the one you shared.

Anyway, to calculate volume you have two solution:

a more performant (but less precise) solution that uses a medium value (e.g. 1k) for each event.

the calculation of volume using the search from license consuming:

if the sourcetypes to monitor ar in a lookup called perimeter.csv:

| tstats count latest(_time) AS lastTime where index=* [| inputlookup perimeter.csv | fields sourcetype ] earliest=-180d latest=now BY host
| eval period=if lastTime>now()-86400*30,"Latest","Previous")
| stats 
     sum(count) AS count 
     dc(period) As period_count
     values(period) AS period
     BY host
| eval status=case(period_count=2,"Always present",period="Latest","Only last Month",period="Previous","Only Previous")
| eval volume=count*1/1024/1024
| table host status volume

if instead you want a more detailed solutions but very less performant, you could try:

index=_internal [ rest splunk_server=local /services/server/info 
    | return host] source=*license_usage.log* type="Usage" 
 [| tstats count latest(_time) AS lastTime where index=* [| inputlookup perimeter.csv | fields sourcetype ] earliest=-180d latest=now BY host | fields host ]
| eval h=if(len(h)=0 OR isnull(h),"(SQUASHED)",h) 
| eval s=if(len(s)=0 OR isnull(s),"(SQUASHED)",s) 
| eval idx=if(len(idx)=0 OR isnull(idx),"(UNKNOWN)",idx) 
| bin _time span=1d 
| stats sum(b) as b by _time, pool, s, st, h, idx 
| timechart span=1d sum(b) AS volumeB by h fixedrange=false 
| fields - _timediff 
| foreach "*" 
    [ eval <<FIELD>>=round('<<FIELD>>'/1024/1024/1024, 3)]

Ciao.

Giuseppe

[sverdhan]

thank you for your reply , can you suggest any method apart from the lookup one?

[giuseppe]

.

[ITWhisperer]

Instead of inputlookup, you could use your metadata search to retrieve the sourcetype names. Obviously, this will only include the sourcetypes for which there have been events, and not for all configured sourcetypes.

[sverdhan]

Thank you , Do you have  a general query to calculate the volume ingested for any sourcetype in general?

[Jawahir]

index=_internal source=*license_usage.log type="Usage" 
| eval indexname = if(len(idx)=0 OR isnull(idx),"(UNKNOWN)",idx)
| eval sourcetypename = st
| bin _time span=1d 
| stats sum(b) as b by _time, pool, indexname, sourcetypename
| eval GB=round(b/1024/1024/1024, 3)
| fields _time, indexname, sourcetypename, GB

[gcusello]

Hi @sverdhan ,

you asked for a list of sourcetypes.

If you want all the sourcetypes, you could try:

index=_internal [ rest splunk_server=local /services/server/info 
    | return host] source=*license_usage.log* type="Usage" 
| eval h=if(len(h)=0 OR isnull(h),"(SQUASHED)",h) 
| eval s=if(len(s)=0 OR isnull(s),"(SQUASHED)",s) 
| eval idx=if(len(idx)=0 OR isnull(idx),"(UNKNOWN)",idx) 
| bin _time span=1d 
| stats sum(b) as b by _time, pool, s, st, h, idx 
| timechart span=1d sum(b) AS volumeB by h fixedrange=false 
| fields - _timediff 
| foreach "*" 
    [ eval <<FIELD>>=round('<<FIELD>>'/1024/1024/1024, 3)]

that's the one that you can find in the license consuming.

Ciao.

Giuseppe