topic Re: Logs not coming to splunk from UF in Getting Data In

Logs not coming to splunk from UF

Hemant93 — Fri, 30 Jun 2023 11:47:10 GMT

Hi team,

Logs are not coming to splunk .The UF is working fine and even connected to indexers, inputs.conf and everything seems perfect.
we are facing this issue for few UFs only.
can you suggest something which i should check?

These are the warnings we are getting :-

1. Search peer dallpspiap090m has the following message: Daily indexing volume limit exceeded. Per the Splunk Enterprise license policy in effect, search is disabled after 5 warnings over a 30-day window. Your Splunk deployment is subject to license enforcement. See License Manager for details.

2. Root Cause(s):

Sum of 3 highest per-cpu iowaits reached red threshold of 15
Sum of 3 highest per-cpu iowaits reached yellow threshold of 7
Maximum per-cpu iowait reached red threshold of 10
- Unhealthy Instances:
  - dallpshdap010m
  - mialvshdap010m.vtitel.net
  - dallvissap010m.vtitel.net
  - mialvissap030m.vtitel.net
  - dallvissap030m.vtitel.net
  - mialvissap010m.vtitel.net
  - dallvissap020m.vtitel.net
  - mialvissap020m.vtitel.net
    3. Search Lag
    Root Cause(s):
    The percentage of non high priority searches lagged (67%) over the last 24 hours is very high and exceeded the yellow thresholds (40%) on this Splunk instance. Total Searches that were part of this percentage=268303. Total lagged Searches=182113

Re: Logs not coming to splunk from UF

isoutamo — Fri, 30 Jun 2023 11:51:17 GMT

your logs are coming to splunk, but you cannot search those as you are ingested too many times over your license quota.

Search peer dallpspiap090m has the following message: Daily indexing volume limit exceeded. Per the Splunk Enterprise license policy in effect, search is disabled after 5 warnings over a 30-day window. Your Splunk deployment is subject to license enforcement. See License Manager for details.

You need to order Unlock license from Splunk. Contact to your account team and ask this.

r. Ismo

Re: Logs not coming to splunk from UF

Hemant93 — Fri, 30 Jun 2023 12:03:18 GMT

Hi Isoutamo,

But we are getting for most of the servers but not getting logs for recently configured servers.

Re: Logs not coming to splunk from UF

PickleRick — Fri, 30 Jun 2023 16:53:13 GMT

This is interesting since the license warning says about 5 violations during 30-day period which is the typical setting for a Splunk Free instance. Your environment seems much bigger than the one for Splunk Free instance.

There is probably more things wrong underneath.

We don't know your event routing, we don't know your architecture, we don't know your search settings.

I'd advise you get a consultant to look over your environment because it looks as if you have more problems than just events which are supposedly not showing in search (but they might be although they might be wrongly parsed and misplaced, for example).

Re: Logs not coming to splunk from UF

isoutamo — Fri, 30 Jun 2023 18:39:25 GMT

Definitely there seems to be something else too. 5/30 was normal limit with older 7&8 versions, not only free. If your instance is using free license then you cannot get unlock license. That’s just for paid customers!

topic Re: Logs not coming to splunk from UF in Getting Data In

Logs not coming to splunk from UF

3. Search Lag

Re: Logs not coming to splunk from UF

Re: Logs not coming to splunk from UF

Re: Logs not coming to splunk from UF

Re: Logs not coming to splunk from UF