Calculate historical ingestion from a specific set of hosts to index over time

[JohnEGones]

Hi Guys (and Gals),

Hopefully quick question, and it's late, so my brain isn't firing quickly/properly.

I need to run a query to get the ingestion over time over two variables: host, index

In the specific case, need to determine if the data ingestion from a specific set of hosts, and whether the data inbound has been increasing more than normally expected. 

So the query would look like:

index=linuxos host IN (server1, server2, server3...) [or possibly you may have a lookup of the set of hosts]
| eval sum(the data per host over hour {or whatever regular chunk of time you want} for a 7 day period)
| timechart xyz

==> chart over a line graph

Also, if there is relevant dashboard/console in the monitoring console I am not thinking of please direct me to the relevant menu or docs.

Appreciate any assistance.

[JohnEGones]

I am trying to run this query but Splunk is complaining that the eval is malformed.

https://docs.splunk.com/Documentation/SCS/current/SearchReference/EvalCommandExamples

I am not sure from the docs how to try to fix this. 

[PickleRick]

Your link leads to wrong documentation (but for some strange reason Google seems to favour it over the proper SPL documentation). There are two different search languages - SPL and SPL2. SPL is used within Splunk Enterprise (and Splunk Cloud), SPL2 is used here and there (I think most notable use is the Edge Processor) but it's not as widely used as SPL. I know it's confusing 😉

Anyway, you need docs for SPL, not SPL2.

https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/WhatsInThisManual

[PickleRick]

While @bowesmana 's solution is correct, it might not be the fastest one 🙂

If your data haven't already rolled over past retention date, you can see if the licensing report is enough for you (but as far as I remember it's either by host or by index).

Unfortunately, if you want to measure the size of raw data (which is what you're asking about), you need to read all the raw data back from the time period you need to analyze. Which is gonna be painfully slow if your environment is of any decent size.

[bowesmana · Accepted solution]

You can do

index=* 
| eval group=index.":".host
| timechart span=1h sum(eval(len(_raw))) as len by group

Use subsearches with lookups to determine which index / host set you want to restrict to

Note with timechart, it will limit the number of groups to 10, so use limit=X where X is the number of index/host pairs to watch