topic Re: How to sort source based on its earliest event indexing? in Splunk Search

How to sort source based on its earliest event indexing?

devd25 — Tue, 17 Oct 2017 07:42:52 GMT

I am in the log sources provisioning phase.

I examine the "data summary" frequently to see the change in number of hosts/sources/sourcetypes to determine from which log sources, Splunk has started collecting/receving data

However, now I have noticed jump in number of sources but same number of hosts and sourcetypes. Hence, I want to be able to find out which was that new source that has newly emerged in Splunk.

In order to do this, I am looking for a search command that will give me a list of all sources with it's first event displayed which, I guess, can be achieved by using the earliest event command.

Can someone please advise how I can achieve this ?

Re: How to sort source based on its earliest event indexing?

gcusello — Tue, 17 Oct 2017 08:45:55 GMT

Hi devd25,
try something like this

index=*
| stats earliest(_time) AS earliest count by source host

to have, for each host and source the first event timestamp and the count of events.

Bye.
Giuseppe

Re: How to sort source based on its earliest event indexing?

devd25 — Wed, 18 Oct 2017 03:02:37 GMT

Hi @cusello,

Thanks for that. I applied that command but it only showed limited hosts, not all of them.

Re: How to sort source based on its earliest event indexing?

gcusello — Wed, 18 Oct 2017 06:54:30 GMT

Hi devd25,
it's strange! this means that you have logs only from the listed hosts, if you're waiting for logs from all your hosts probably there's a problem in ingestion or maybe some logs arrive only in certain time periods different from the ones you used in your search.

You can list all connected hosts using the same search with Splunk internal logs:

index=_internal
| stats earliest(_time) AS earliest count by source host

Check if the result of this search is different from the previous, so you can know the host to verify in detail.

One of the usual alert is just the perimeter check, creating a lookup with all the hosts (Perimeter.csv with one coluimn called host) to monitor and running every 5 minutes a search like this

index=_internal
| eval host=upper(host)
| stats  count by host
| append [ | inputlookup perimeter.csv | eval host=upper(host), count=0 | fields host ]
| stats sum(count) AS Total by host
| where Total=0

if there are results to this search there's a connection problem for the resulted hosts.

Bye.
Giuseppe