Logs not coming to splunk from UF

[Hemant93]

Hi team,

Logs are not coming to splunk .The UF is working fine and even connected to indexers, inputs.conf and everything seems perfect.
we are facing this issue for few UFs only.
can you suggest something which i should check? 

These are the warnings we are getting :- 

1. Search peer dallpspiap090m has the following message: Daily indexing volume limit exceeded. Per the Splunk Enterprise license policy in effect, search is disabled after 5 warnings over a 30-day window. Your Splunk deployment is subject to license enforcement. See License Manager for details.

2. Root Cause(s):

• Sum of 3 highest per-cpu iowaits reached red threshold of 15
• Sum of 3 highest per-cpu iowaits reached yellow threshold of 7
• Maximum per-cpu iowait reached red threshold of 10
  • Unhealthy Instances:

    • dallpshdap010m
    • mialvshdap010m.vtitel.net
    • dallvissap010m.vtitel.net
    • mialvissap030m.vtitel.net
    • dallvissap030m.vtitel.net
    • mialvissap010m.vtitel.net
    • dallvissap020m.vtitel.net
    • mialvissap020m.vtitel.net

       3. Search Lag

      • Root Cause(s):
        • The percentage of non high priority searches lagged (67%) over the last 24 hours is very high and exceeded the yellow thresholds (40%) on this Splunk instance. Total Searches that were part of this percentage=268303. Total lagged Searches=182113
          
          
          

[PickleRick]

This is interesting since the license warning says about 5 violations during 30-day period which is the typical setting for a Splunk Free instance. Your environment seems much bigger than the one for Splunk Free instance.

There is probably more things wrong underneath.

We don't know your event routing, we don't know your architecture, we don't know your search settings.

I'd advise you get a consultant to look over your environment because it looks as if you have more problems than just events which are supposedly not showing in search (but they might be although they might be wrongly parsed and misplaced, for example).

[isoutamo]

Definitely there seems to be something else too. 5/30 was normal limit with older 7&8 versions, not only free. If your instance is using free license then you cannot get unlock license. That’s just for paid customers!

[isoutamo]

Hi

your logs are coming to splunk, but you cannot search those as you are ingested too many times over your license quota.

 Search peer dallpspiap090m has the following message: Daily indexing volume limit exceeded. Per the Splunk Enterprise license policy in effect, search is disabled after 5 warnings over a 30-day window. Your Splunk deployment is subject to license enforcement. See License Manager for details.

You need to order Unlock license from Splunk. Contact to your account team and ask this.

r. Ismo 

[Hemant93]

Hi Isoutamo,

But we are getting for most of the servers but not getting logs for recently configured servers.