Heartbeat Tracker

[majilan1]

Hi Splunkers,

I have a question and I need help from experts, I'm working on creating a heartbeat tracker search that monitor when a host gets span up, and it's a window or Linux it gets generic apps from the server class, so there is a server class built out there that is just looking for any host that isn't already in the server class.

So the purpose of the heartbeat tracker is to inform us that there is a brand-new host that isn't in the server class, so the ask is to track the hosts that showing up in the heartbeat index and if these hosts are there for multiple days that means they need to be addressed, as an example every host that get span up whether we know about it or not is going to get the heartbeat initially, so it's going to span up, and it's going to get the heartbeat and once it's get to its real app it's going to stop sending logs to the heartbeat index, so what I really want to know is per host how many days has it been talking to the X index so if I get a host that has been talking to the X index for several days then I know that isn't the initial start up, it's a problem that need to be looked at.

| tstats count where index=X by host index span=1d _time

[PickleRick]

Ok. Firstly, invest in some punctuation, please, because this stream of conciousness is dificult to read.

Secondly, what are you spinning up? You mention server classes so I suspect you're talking about creating some (virtual? Doesn't matter really) machines with a pre-installed UF.

And now what? That UF contains some pre-defined setting, especially including outputs.conf? If it does, then what do you want do "heartbeat"? It's gonna be sending its own internal logs anyway.

It is also a fairly typical practice to distribute with your UF a kind of a "starter pack" of standard apps containing common configuration items (like DS address, outputs.conf and such) and generally accept all hosts to a serverclass distributing current versions of those apps.

So what heartbeat do you want?

[majilan1]

Sorry for the punctuation problem. What I simply need is just to create a search that say for every host tell me how many days it's been talking to the index X, and use a command that does a span=1d.

The heartbeat to monitor the hosts to make sure they are going to the right place. This often done by sending or tracking periodic signals from the host to Splunk. 

Thanks for your help and efforts.

[PickleRick]

If you simply want to find out when a host started sending data to index you simply need to find min(_time).

| tstats min(_time) where index=something earliest=1 by host

Two caveats

1) It's based on the _time field so if you've ingested a backlog of 3 years worth of data right after deploying your forwarder, your results will probably not be true. I don't remember if you can use _indextime in tstats. You have to check.

2) It will of course only show data from the buckets which haven't yet rolled to frozen so for old data it will not be true