This is a really old post but I had the same problem.  A search query that appears to be helping me find these problems is: index=_internal sourcetype=splunkd log_level=ERROR component=HttpInputDataHandler The results are imperfect because they don't exactly match what's shown in the authentication failures, but in my case, it appears the errors are being caused by a source that is sending in blank/missing tokens.

I was not getting any data together, that was what was wrong... sorry for the miscommunication. I implemented your idea, and I am getting data now! Thank you!!! However, I am not getting the  Employee column filled. It might because of the data issue. But, I wanted to know if we can label the fields by source. For example. I have UserNumber in both sources that mean different things and name in both sources that mean different things... How can I help Splunk differentiate them? Is there any resources would you suggest? Thank you so much!

It depends what it is you are trying to do, and what you think is wrong. As it stands, PARENT_ACCOUNT is not a field beyond the stats command (since it isn't listed as an output field - dc just counts the distinct values of the field without listing them). For the "join", you don't need a join (and they usually should be avoided if possible as they are slow and have limitations). Try something like this: index=* sourcetype=transaction OR sourcetype=users | eval USERNUMBER=coalesce(USERNUMBER, NUMBER) | eventstats values(NAME) as Employee by USERNUMBER | stats dc(PARENT_ACCOUNT) as transactionMade values(Employee) as Employee by POSTDATE, USERNUMBER | table USERNUMBER Employee PARENT_ACCOUNT POSTDATE transactionMade

We have upwards of 250k forwarders in one of our environments and various levels of DNS caching that make it very difficult for a forwarder to request a deployment server IP from a DNS name and maintain the connection consistently in order for it to get appropriate apps downloaded.  I have seen where a system will request an IP from a DNS name, make an initial connection to a deployment server, then send a DNS query again only to be given a different IP address, which causes issues with the forwarder trying to establish a consistent trusted connection to a deployment server.  That switch in deployment server destinations causes the forwarder to just try again later, until it can establish a consistent connection randomly. We put our deployment servers behind a load balancer before, but all the connections and logs show the forwarders coming from the same ip address.. something x-forwarded-for should help solve at our scale.

Hi @parthiban, you hughlighted only the 403 response code, if you want the full string, you could use: | rex "\"processing_stage\": \"(?<response>[^\"]+)" that you can test at https://regex101.com/r/mz4c1L/2  Ciao. Giuseppe