Why my Splunk ingestion has decreased?

[IAskALotOfQs]

Hi all, I have been looking at my Splunk CMC for a customer and have noticed that the ingest per day has been up and down since early November, I have had a look at the CMC (cloud monitoring console) but for some tabs, the graphs shown by default won't let me go back to November to find trends such as "daily event count per day in November"

Could someone guide me on why this is & what would be a good place to start on this investigation. For context:

Arch is:

UF --> HF --> SC

SC4S --> SC

Cloud data --> HF --> SC

[richgalloway]

Fluctuations in ingest are normal.  If what you're seeing appears abnormal, then there are a few things to check.

1) Verify the UF and SC4S are still running.

2) Restart the UF and/or SC4S

3) Confirm the applications generating the data are still running.

4) Check for any network changes that may be blocking ingestion.

5) Check the UF and SC4S logs to see if they're reporting any problems sending data.

6) Confirm the certificates used (if any) have not expired.

The data used by the CMC to show ingestion rates is retained for only 30 days by default. That is why you cannot view the rates for November.

[IAskALotOfQs]

Thanks for the reply, the customer is seeing fluctuations from at its peak was 4TB to now around 2.8 TB. I am in a prod environment so can not restart as there would to be too much emailing and authorising to comply with.

What would be a good way to investigate this/ some graphs to indicate if there has been a decrease in events/stayed the same, or if there has been a decrease in the thruput (would this be relevant as I'd need to know the volume of data just before it's indexed and counted to licesning meter PER INDEX)

[richgalloway]

The CMC may have dashboard panels showing ingestion by index and/or sourcetype.  You can get similar information yourself with a search like this.  It counts events rather than TB for better performance, but should offer good insights.

| tstats prestats=t count where index=foo sourcetype=bar by _time span=1d
| timechart span=1d count

[PickleRick]

Yep. tstats and timechart is a good way to go about finding out if you have some significant changes in your event counts (as opposed to normal daily/weekly variance). You can draw yourself a nice line/bar chart and easily see visually if your event rates are changing.

You can use different aggregations to investigate it further (by sourcetype, by source, by host...).

Typically event size distribution should not change much unless there has been some change on the source's side (but if you have many different sources, such change on just one or two sources would not reflect much on overall data rate unless of course you have a single source "dominating" in your data).