Spike in log ingestion only on weekends

[ayoungUSU]

Hello,

Can anyone assist in determining why my splunk instance ingest large amounts of data ONLY on the weekends?  This appears to be across the board for all hosts as near as I can tell.  

I run this command:

index=_internal metrics kb series!=_* "group=per_host_thruput" earliest=-30d | eval mb = kb / 1024 | timechart fixedrange=t span=1d sum(mb) by series

and it shows the daily ingest for numerous forwarders.  During the week it averages out but over the weekend it exceeds my daily ingest limit causing warnings. 

I would like to be able to find out what the cause is and a possible solution so I can even out the ingestion so I dont get violations.

Much appreciated for any assistance!

[richgalloway]

Try variations on your query to see if you isolate the source or sourcetype that is causing the spike.

index=_internal metrics kb series!=_* "group=per_source_thruput" earliest=-30d 
| eval mb = kb / 1024 
| timechart fixedrange=t span=1d sum(mb) by series

index=_internal metrics kb series!=_* "group=per_index_thruput" earliest=-30d 
| eval mb = kb / 1024 
| timechart fixedrange=t span=1d sum(mb) by series

[ITWhisperer]

This is probably down to your data sources. You should check for patterns of increased logging by your apps at the weekends, other activity on the hosts, etc. Can you narrow down the time periods when the increase in logging occurs? Do you have any batch jobs running at these times which might account for the additional data? You need to investigate the nature of the increase further.