topic How to detect missing log sources/Internal logs in Splunk? in Alerting

How to detect missing log sources/Internal logs in Splunk?

potnuru — Wed, 24 Jun 2020 18:28:33 GMT

Please help me in detecting the below scenarios for alerting.

1) If a UF stops forwarding the actual source logs (Example: Windows Event Logs ) but it is forwarding the _internal logs.
2) if a UF stops forwarding actual source logs and _internal logs.
3) How to find whether the UF is reporting to Deployment Server from Search Head.

Also, please let me know the solution/process for the above if
a) HF is configured in the Splunk Environment.
b) When HF is not configured in the Splunk Environment.

I have checked the app UFMA app in Splunk Base. But I don't know whether it can fulfill my needs as I can't make deployment server as a search peer which is required for the UFMA app to function.

Re: How to detect missing log sources/Internal logs in Splunk?

gcusello — Wed, 30 Sep 2020 04:27:58 GMT

Hi @potnuru,
some little clarifications:

are item 1 and 2 in the same search or in different ones?
what do you mean in item3: to know if the UF is listed in deployment clients list of Deployment Server or what else?
what do you mean in the second part of your question ("configured")?

Anyway the first two items could be solved in this way:

create a lookup (called e.g. perimeter.csv) containing the list of the monitored hosts (at least one column called e.g. host);
run a search like this:

| metasearch index=_internal OR index=*
| eval host=lower(host), type=if(index=_internal,"Internal","Other Indexes")
| stats values(type) AS type count BY host
| append [ | inputlookup perimeter.csv | eval host=lower(host), count=0, type="Not Present" | fields host count type ]
| stats values(type) AS type sum(count) AS total BY host
| eval Status=if(type="Not Present","Not Present", if(type="Internal","Internal","Both Internal and Others"))
| table host Status

Ciao.
Giuseppe

Re: How to detect missing log sources/Internal logs in Splunk?

potnuru — Thu, 05 Mar 2020 04:44:17 GMT

Hi @gcusello

Thank you for your response.

The Item 1 and Item 2 are two different scenarios and I want to create a single Dashboard for both of these where I want to show them in same Panel with different colors or severity.

Item 3 is to detect the last phone home time of a UF from search head and not on Deployment server.

The second part of my question is 1
a) what will be the query if HF is used between UF and Indexer.
b) if HF is not present between UF and Indexer.

Basically I want to know if there is any difference in the query if HF is present in between UF and Indexer.

Re: How to detect missing log sources/Internal logs in Splunk?

gcusello — Thu, 05 Mar 2020 07:41:27 GMT

Hi @potnuru,
so does my answer solve the first two questions or do you want in the same panel also the info of last connection?
Anyway to add also this info to the same panel, you cannot use the | metasearch command so your search will be slower.
Anyway you have to add to the stats commands an option with the latest _time, something like this:

index=_internal OR index=*
| eval host=lower(host), type=if(index=_internal,"Internal","Other Indexes")
| stats values(type) AS type latest(_time) AS latest count BY host
| append [ | inputlookup perimeter.csv | eval host=lower(host), count=0, type="Not Present" | fields host count type ]
| stats values(type) AS type values(latest) AS latest sum(count) AS total BY host
| eval Status=if(type="Not Present","Not Present", if(type="Internal","Internal","Both Internal and Others"))
| table host Status latest

About the second part, I cannot help you, for my knowledge, there isn't any way to understand if there's or not an HF between UFs and INDs, I hope that someone else denies me.
As a workaround, you could add to the perimeter.csv lookup a field with this information and use it in the search.

Ciao.
Giuseppe

Re: How to detect missing log sources/Internal logs in Splunk?

potnuru — Fri, 06 Mar 2020 10:07:03 GMT

Thank you @gcusello for your help.

From the above query, I can understand that the last log received for the index is shown in the table as latest.

Is this query is same even if we use the HF in between UF and Indexer?

Also, how can we get the last deployment server Phone Home time in Search Head?

Re: How to detect missing log sources/Internal logs in Splunk?

gcusello — Fri, 06 Mar 2020 12:02:56 GMT

Hi @potnuru,
the presence of an HF doesn't change the search because data starts from UFs and arrive to Indexers where a I search using a Search Head: an intermediate HF doesn't change the data (for this reason I cannot know if there's an intermediate HF in a data.
Really HF parse the data and send coocked data to Indexers but is doesn't add any information to understand this.

About the last Deployment Server Phone Home, you could search

index=_internal host=your_deployment_server "Running phone"
| stats latest(_time) AS latest BY hostname

Ciao.
Giuseppe

Re: How to detect missing log sources/Internal logs in Splunk?

potnuru — Fri, 06 Mar 2020 12:40:06 GMT

Thank you @gcusello for being so helpful.

Re: How to detect missing log sources/Internal logs in Splunk?

gcusello — Fri, 06 Mar 2020 12:46:26 GMT

You're welcome!
Ciao and next time.
Giuseppe