topic Re: Quickest way to ensure data is coming in for a sourcetype across dozens of servers... in Getting Data In

Quickest way to ensure data is coming in for a sourcetype across dozens of servers...

a212830 — Mon, 11 Apr 2016 01:46:47 GMT

Hi,

I have an app that creates lots of files (roll over at 50mb, about every 2-3 min during business hours), and has lots of servers (50+). I've had complaints that data and/or files are missing on occasion, so I'm looking for a quick/efficient way to ensure these servers and sourcetype are continuously sending data into Splunk and if they aren't I can identify the server having an issue.

Any suggestions?

Re: Quickest way to ensure data is coming in for a sourcetype across dozens of servers...

MuS — Mon, 11 Apr 2016 03:37:53 GMT

Hi a212830,

if you are on Splunk 6.2.x and newer, you can use the Distributed Management Console http://docs.splunk.com/Documentation/Splunk/latest/DMC/WhatcanDMCdo which has pre-defined searches to cover that.

Another command is metadata http://docs.splunk.com/Documentation/Splunk/6.4.0/SearchReference/Metadata which will give you a quick overview, use it like this:

 | metadata type=sourcetypes index=*

Hope this helps ...

cheers, MuS

Re: Quickest way to ensure data is coming in for a sourcetype across dozens of servers...

martin_mueller — Mon, 11 Apr 2016 21:48:41 GMT

Additionally, make sure your forwarders also monitor rolled .1, .2, etc. files in case there is a short congestion. Don't monitor rolled .gz files though.

Re: Quickest way to ensure data is coming in for a sourcetype across dozens of servers...

a212830 — Fri, 15 Apr 2016 14:33:54 GMT

metadata could be useful, but how does one track it across sources and hosts - doesn't seem to be possible.

Re: Quickest way to ensure data is coming in for a sourcetype across dozens of servers...

jplumsdaine22 — Fri, 15 Apr 2016 16:38:41 GMT

metasearch is another useful command for this sort of thing if you don't have a DMC running

| metasearch sourcetype=<your_sourcetype>  | timechart useother=f values(source) by host

This search should show you which sources are indexed per host (adjust your span= for timechart to suit). If you don't have many different filenames I would probably swap the aggregator:

| metasearch sourcetype=<your_sourcetype>  | timechart useother=f values(host) by source

You can also just do a distinct count and set an alert based on how many hosts there should be.

| metasearch sourcetype=<your_sourcetype>  | timechart useother=f dc(host) by source

Checkout http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Metasearch

Re: Quickest way to ensure data is coming in for a sourcetype across dozens of servers...

martin_mueller — Fri, 15 Apr 2016 23:15:20 GMT

In any imaginable scenario except "I don't know of tstats", use |tstats instead of |metadata:

| tstats count where index=* by host sourcetype source

| tstats count where index=* host=suspicious sourcetype=weird by _time source | timechart sum(count) by source

Run the latter over a suspicious time range and see if a stacked bar chart has gaps.

Re: Quickest way to ensure data is coming in for a sourcetype across dozens of servers...

a212830 — Tue, 19 Apr 2016 14:00:19 GMT

Good stuff! Thanks!

Re: Quickest way to ensure data is coming in for a sourcetype across dozens of servers...

jplumsdaine22 — Wed, 20 Apr 2016 14:56:00 GMT

MIND BLOWN

tstas 4 ever

Re: Quickest way to ensure data is coming in for a sourcetype across dozens of servers...

sloshburch — Mon, 13 Jun 2016 20:03:01 GMT

Correct, I can't think of a way to show unique tuples with metadata. You can only show a list of one or the other (or hosts) and append them.

Re: Quickest way to ensure data is coming in for a sourcetype across dozens of servers...

MuS — Mon, 13 Jun 2016 20:23:37 GMT

As @martin_mueller suggested use | tstats count where index=* by host sourcetype source instead