All Posts

Find Answers
Ask questions. Get answers. Find technical product solutions from passionate members of the Splunk community.

All Posts

Hello!  Is it possible to implement something like this? I have 300+ devices that send logs to one index. I want to check if there are no logs from the device for more than one minute then execute ... See more...
Hello!  Is it possible to implement something like this? I have 300+ devices that send logs to one index. I want to check if there are no logs from the device for more than one minute then execute an alert. When the device resumed logs then also a warning. And immediately after the warning update the csv file. My search now looks like this: | tstats latest(_time) as lastSeen where index IN("my_devs") earliest=-2m latest=now by host | lookup devs_hosts_names.csv host OUTPUT dev_name | eval dev_name = if(isnotnull(dev_name),dev_name,"unknow host") | eval status = if((now() - lastSeen<=60),"up","down") | eval status = if(isnotnull(lastSeen),status,"unknow") | search NOT [| inputlookup devs_status.csv | fields host dev_name status] | convert ctime(*Seen) | table host dev_name status lastSeen | - At this time of search I would like to trigger an alert for each dev_name and then rewrite (update)  devs_status.csv  But I don't find how it can be realized, I ask for your help. I'm new to splunk and don't understand how much this kind of request is normal for splunk? Thanks.  
It depends on what you are trying to do since this search seems to be the opposite of what you had previously said you were trying to do.
I have below query .how to include into result query.pls advise     Need to include this one into result query | stats values(event) as events by customer | where NOT (events = "s1" AND events = ... See more...
I have below query .how to include into result query.pls advise     Need to include this one into result query | stats values(event) as events by customer | where NOT (events = "s1" AND events = "s2" AND events = "s3") Result query: (index=1 sourcetype="abc" "s1 event received" and "s2 event received" and "s3 event received") OR (index=2 sourcetype="xyz" "created") | rex "(?<e_type>s.) event received for (?<customer>\d+)" | rex "(?<created>created) for (?<customer>\d+)" | stats max(eval(if(e_type="s3",_time, null()))) as last_e_type max(eval(if(created="created", _time, null()))) as created_time dc(e_type) as e_types values(created) as created by customer | addinfo | where e_types=3 AND (created_time-last_e_type > 300 OR (isnull(created_time) AND info_max_time - last_e_type > 300)
| rex "customer (?<customer>\d+)"
How to extract customer number ?
How to extract customer number from the event .pls advise 
Assuming customer and event have already been extracted | stats values(event) as events by customer | where NOT (events = "s1" AND events = "s2" AND events = "s3")
The AND operator works within a single event.  To combine multiple events you need to use an aggregating command.  Assuming the customer number has been extracted into a field called "customer" then ... See more...
The AND operator works within a single event.  To combine multiple events you need to use an aggregating command.  Assuming the customer number has been extracted into a field called "customer" then this will trigger an alert if any customer does not have all three events. <<some search for S1, S2, and S3>> | stats count by customer | where count < 3  
I have below message in the splunk log   Ex : s1 event has been received for customer 15778 S2 event has been received for customer 15778 S3 event has been received for customer 15778   I want ... See more...
I have below message in the splunk log   Ex : s1 event has been received for customer 15778 S2 event has been received for customer 15778 S3 event has been received for customer 15778   I want to check all S1,S2,S3 event has been received message present in the particular customer.i used AND condition but not able to achieve.plesse help me on this. As per my scenario,if i have 1 lakhs customer, i want to check for all 3 events has been received mesage is present in the splunk log for one particular customer.if not present all 3 mesage i need to set up alert.
Try including the _time as well in your search  Either using timechart or by _time bucket  index=_internal source=*license_usage.log* type=Usage idx=*| eval GB=b/1024/1024/1024 |timchart span=1d s... See more...
Try including the _time as well in your search  Either using timechart or by _time bucket  index=_internal source=*license_usage.log* type=Usage idx=*| eval GB=b/1024/1024/1024 |timchart span=1d sum(gb) by st   or index=_internal source=*license_usage.log* type=Usage idx=*| eval GB=b/1024/1024/1024 |bin _time span=1d | stats sum(GB) by _time,st
I am not sure why you are not adapting my previous suggestion - try something like this | stats count(eval(like(message, "%READ-ERROR -&gt; DP is temporarily down%"))) as read_error_count count(Code... See more...
I am not sure why you are not adapting my previous suggestion - try something like this | stats count(eval(like(message, "%READ-ERROR -&gt; DP is temporarily down%"))) as read_error_count count(Code) as count_by_id |eval read_error_count = if(read_error_count &gt; count_by_id/2 ,mvappend(read_error_count,"RED"), read_error_count)  
| stats count(eval(like(message, %READ-ERROR -&gt; DP is temporarily down%))) as read_error_count |stats count(Code) as count_by_id |eval color_value = if(read_error_count &gt; count/2 ,0,1) i want ... See more...
| stats count(eval(like(message, %READ-ERROR -&gt; DP is temporarily down%))) as read_error_count |stats count(Code) as count_by_id |eval color_value = if(read_error_count &gt; count/2 ,0,1) i want color read_error_count column by color_value result thanks @ITWhisperer 
Correct, the question is not identical, but the essence of the question is, that is, how to colour cells based on the value in another cell on the same row, in your case you want the read_error_count... See more...
Correct, the question is not identical, but the essence of the question is, that is, how to colour cells based on the value in another cell on the same row, in your case you want the read_error_count colour based on the value of the count_by_id. To do this, as I said earlier, and is shown in more detail in the referenced solution, you use mvappend to add a second value to the read_error_coiunt field based on the value of the count_by_id field | eval read_error_count=if(count_by_id > 10, mvappend(read_error_count,"RED"), read_error_count) Then use CSS to hide the second multi-value cell and set the palette to change the colour if "RED" is a value You can use other strings and colours to your requirements.
When I install TA-Demisto App in single Addhoc SH node, it works fine, but when install in using SH Deployer for SH Cluster nodes, does not work and gives me Error:   Configuration page failed to l... See more...
When I install TA-Demisto App in single Addhoc SH node, it works fine, but when install in using SH Deployer for SH Cluster nodes, does not work and gives me Error:   Configuration page failed to load, the server reported internal errors which may indicate you do not have access to this page. Error: Request failed with status code 500 ERR0002 ==> Any idea how to fix that error ?
I have installed events service in same EC, controller and EUM node for demo purpose but after event service installation the service shows as critical in and then stopped.  Please advise what could... See more...
I have installed events service in same EC, controller and EUM node for demo purpose but after event service installation the service shows as critical in and then stopped.  Please advise what could the reason due for this. I have installed the platform componets on windows server 2019 os.  [2023-12-03T12:11:14,393+03:00]  [ERROR]  [dw-58 - PUT /entitysearch/sync]  [c.a.a.s.r.e.j.m.UnknownExceptionMapper]  Unknown exception occurred while processing HTTP request. logCorrelationId=[946fed22-05db-4370-aa1f-3ce469cb4689] [{}] java.lang.RuntimeException: java.net.ConnectException at com.appdynamics.analytics.processor.entitysearch.store.EntitySearchElasticStore.sync(EntitySearchElasticStore.java:114) at com.appdynamics.analytics.processor.entitysearch.resource.EntitySearchResourceImpl.sync(EntitySearchResourceImpl.java:112) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167) at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$VoidOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:159) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79) at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80) at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:255) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:244) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234) at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680) at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394) at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656) at io.dropwizard.servlets.ThreadNameFilter.doFilter(ThreadNameFilter.java:35) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) at io.dropwizard.jersey.filter.AllowedMethodsFilter.handle(AllowedMethodsFilter.java:47) at io.dropwizard.jersey.filter.AllowedMethodsFilter.doFilter(AllowedMethodsFilter.java:41) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) at com.appdynamics.common.framework.util.ClickJackSecurityFilter.doFilter(ClickJackSecurityFilter.java:91) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:313) at io.dropwizard.jetty.RoutingHandler.handle(RoutingHandler.java:52) at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:722) at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:54) at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:181) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.Server.handle(Server.java:516) at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.ConnectException: null at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:930) at org.elasticsearch.client.RestClient.performRequest(RestClient.java:300) at org.elasticsearch.client.RestClient.performRequest(RestClient.java:288) at co.elastic.clients.transport.rest_client.RestClientTransport.performRequest(RestClientTransport.java:147) at co.elastic.clients.elasticsearch.ElasticsearchClient.bulk(ElasticsearchClient.java:319) at co.elastic.clients.elasticsearch.ElasticsearchClient.bulk(ElasticsearchClient.java:336) at com.appdynamics.analytics.processor.entitysearch.store.EntitySearchElasticStore.sync(EntitySearchElasticStore.java:107) ... 70 common frames omitted Caused by: java.net.ConnectException: null at org.apache.http.nio.pool.RouteSpecificPool.timeout(RouteSpecificPool.java:168) at org.apache.http.nio.pool.AbstractNIOConnPool.requestTimeout(AbstractNIOConnPool.java:561) at org.apache.http.nio.pool.AbstractNIOConnPool$InternalSessionRequestCallback.timeout(AbstractNIOConnPool.java:822) at org.apache.http.impl.nio.reactor.SessionRequestImpl.timeout(SessionRequestImpl.java:183) at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processTimeouts(DefaultConnectingIOReactor.java:210) at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:155) at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:348) at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:191) at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64) ... 1 common frames omitted [2023-12-03T12:11:30,392+03:00]  [ERROR]  [dw-58 - GET /v1/elasticsearch/numberOfDataNodes]  [c.a.a.s.r.e.j.m.UnknownExceptionMapper]  Unknown exception occurred while processing HTTP request. logCorrelationId=[a3823670-1430-4185-b355-b5c3be28244e] [{}] java.lang.RuntimeException: java.net.ConnectException at com.appdynamics.analytics.elasticsearch.client.ElasticSearchClientFacade.getDetailedClusterHealth(ElasticSearchClientFacade.java:388) at com.appdynamics.analytics.processor.elasticsearch.settings.ElasticSearchSettingsResource.getNumDataNodes(ElasticSearchSettingsResource.java:77) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167) at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79) at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80) at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:255) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:244) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234) at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680) at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394) at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656) at io.dropwizard.servlets.ThreadNameFilter.doFilter(ThreadNameFilter.java:35) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) at io.dropwizard.jersey.filter.AllowedMethodsFilter.handle(AllowedMethodsFilter.java:47) at io.dropwizard.jersey.filter.AllowedMethodsFilter.doFilter(AllowedMethodsFilter.java:41) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) at com.appdynamics.common.framework.util.ClickJackSecurityFilter.doFilter(ClickJackSecurityFilter.java:91) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:313) at io.dropwizard.jetty.RoutingHandler.handle(RoutingHandler.java:52) at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772) at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:54) at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:181) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.Server.handle(Server.java:516) at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.ConnectException: null at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:930) at org.elasticsearch.client.RestClient.performRequest(RestClient.java:300) at org.elasticsearch.client.RestClient.performRequest(RestClient.java:288) at co.elastic.clients.transport.rest_client.RestClientTransport.performRequest(RestClientTransport.java:147) at co.elastic.clients.elasticsearch.cluster.ElasticsearchClusterClient.health(ElasticsearchClusterClient.java:334) at com.appdynamics.analytics.elasticsearch.admin.ElasticSearchAdminFacade.getDetailedClusterHealth(ElasticSearchAdminFacade.java:122) at com.appdynamics.analytics.elasticsearch.client.ElasticSearchClientFacade.getDetailedClusterHealth(ElasticSearchClientFacade.java:386) ... 70 common frames omitted Caused by: java.net.ConnectException: null at org.apache.http.nio.pool.RouteSpecificPool.timeout(RouteSpecificPool.java:168) at org.apache.http.nio.pool.AbstractNIOConnPool.requestTimeout(AbstractNIOConnPool.java:561) at org.apache.http.nio.pool.AbstractNIOConnPool$InternalSessionRequestCallback.timeout(AbstractNIOConnPool.java:822) at org.apache.http.impl.nio.reactor.SessionRequestImpl.timeout(SessionRequestImpl.java:183) at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processTimeouts(DefaultConnectingIOReactor.java:210) at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:155) at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:348) at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:191) at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64) ... 1 common frames omitted [2023-12-03T12:11:31,443+03:00]  [ERROR]  [dw-59 - GET /v1/elasticsearch/numberOfDataNodes]  [c.a.a.s.r.e.j.m.UnknownExceptionMapper]  Unknown exception occurred while processing HTTP request. logCorrelationId=[54da9d56-e317-4276-8488-8358eb796a14] [{}] java.lang.RuntimeException: java.net.ConnectException at com.appdynamics.analytics.elasticsearch.client.ElasticSearchClientFacade.getDetailedClusterHealth(ElasticSearchClientFacade.java:388) at com.appdynamics.analytics.processor.elasticsearch.settings.ElasticSearchSettingsResource.getNumDataNodes(ElasticSearchSettingsResource.java:77) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167) at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79) at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80) at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:255) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:244) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234) at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680) at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394) at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656) at io.dropwizard.servlets.ThreadNameFilter.doFilter(ThreadNameFilter.java:35) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) at io.dropwizard.jersey.filter.AllowedMethodsFilter.handle(AllowedMethodsFilter.java:47) at io.dropwizard.jersey.filter.AllowedMethodsFilter.doFilter(AllowedMethodsFilter.java:41) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) at com.appdynamics.common.framework.util.ClickJackSecurityFilter.doFilter(ClickJackSecurityFilter.java:91) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:313) at io.dropwizard.jetty.RoutingHandler.handle(RoutingHandler.java:52) at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772) at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:54) at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:181) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.Server.handle(Server.java:516) at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.ConnectException: null at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:930) at org.elasticsearch.client.RestClient.performRequest(RestClient.java:300) at org.elasticsearch.client.RestClient.performRequest(RestClient.java:288) at co.elastic.clients.transport.rest_client.RestClientTransport.performRequest(RestClientTransport.java:147) at co.elastic.clients.elasticsearch.cluster.ElasticsearchClusterClient.health(ElasticsearchClusterClient.java:334) at com.appdynamics.analytics.elasticsearch.admin.ElasticSearchAdminFacade.getDetailedClusterHealth(ElasticSearchAdminFacade.java:122) at com.appdynamics.analytics.elasticsearch.client.ElasticSearchClientFacade.getDetailedClusterHealth(ElasticSearchClientFacade.java:386) ... 70 common frames omitted Caused by: java.net.ConnectException: null at org.apache.http.nio.pool.RouteSpecificPool.timeout(RouteSpecificPool.java:168) at org.apache.http.nio.pool.AbstractNIOConnPool.requestTimeout(AbstractNIOConnPool.java:561) at org.apache.http.nio.pool.AbstractNIOConnPool$InternalSessionRequestCallback.timeout(AbstractNIOConnPool.java:822) at org.apache.http.impl.nio.reactor.SessionRequestImpl.timeout(SessionRequestImpl.java:183) at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processTimeouts(DefaultConnectingIOReactor.java:210) at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:155) at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:348) at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:191) at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64) ... 1 common frames omitted [2023-12-03T12:11:41,557+03:00]  [WARN ]  [health-report-thread-0]  [c.a.c.u.health.HealthReporterModule]  Report unhealthy [ deadlocks: (healthy)    events-service-api-store / BizOutcome [stage.extract]: (healthy) Rates (Avg per second. Avg of last 5 min) accepted events: [0.000000], discarded events: [0.000000], acceptance ratio: [0.000000]   events-service-api-store / BizOutcome [stage.filter]: (healthy) Rates (Avg per second. Avg of last 5 min) accepted events: [0.000000], discarded events: [0.000000], acceptance ratio: [0.000000]   events-service-api-store / BizOutcome [stage.parse]: (healthy) Rates (Avg per second. Avg of last 5 min) accepted events: [0.000000], discarded events: [0.000000], acceptance ratio: [0.000000]   events-service-api-store / BizOutcome [stage.upsert]: (healthy) Rates (Avg per second. Avg of last 5 min) batch size: [0.000000], success: [0.000000], failure: [0.000000]   events-service-api-store / Build information: (healthy) buildName=release/23.7.0.onprem.next-Analytics-release2370onpremnext-23.7.0-246, buildHash=aee38fe8666e1090e98c850a093335567ad2bad4, buildTimestamp=2023-08-03T12:19:23+0000, buildNumber=23.7.0-246, jobName=Analytics-release2370onpremnext, version=23.7.0-246   events-service-api-store / Configuration properties: (healthy) Dynamic properties: [query.default.pagination.size=1000, query.default.scroll.batch.size=1000, ad.accountmanager.accountConfigCacheExpireSeconds=900, query.funnel.batch.scroll.expiry.time.millis=60000, query.default.pagination.offset=0, ad.accountmanager.key.admin_service=REDACTED, ad.es.event.index.maxBulkUpdateSizeBytes=5000000, query.default.results.limit=100, query.nested.enable=false, query.max.bucket.nesting.level=5, ad.es.event.maxUpsertRequestBodySizeBytes=5000000, ad.accountmanager.key.account_service=REDACTED, ad.es.event.index.maxBulkUpdateNumDocs=200, query.exact.analyzed.wildcard.enable=true, query.validator.max.batch.count=20, query.max.forced.range.size=2000, ad.accountmanager.key.disabledKeys=REDACTED, ad.es.event.index.fieldNumberIncrement=500, ad.accountmanager.cacheSize=5000, query.default.aggregation.level.limit=10, schema.validator.max.custom.events.per.account=20, query.max.pagination.offset=10000, ad.metric.processor.enabled.retry.onMpFailure=true, query.max.pagination.size=10000, query.max.results.series.limit=30000, ad.es.response.filter.double.bucketFunctionsRequiredToConvertKeyToLong=REDACTED, query.max.concat.arguments=100, ad.accountmanager.eumAccountCacheExpireSeconds=3600, query.max.results.limit=10000, query.standard.search.call.timeout.millis=300000, query.max.aggregation.level.limit=100, query.funnel.batch.enable.processing=false, ad.env=dev, ad.feature.adql.454.functions.values={"all": ["now","toDate","toFloat","toInt","toString","ifNull","round"], "none": []}, ad.es.event.index.fieldIncrementThresholdPercentage=80, ad.metric.processor.enabled.clusters=[], query.max.export.threads=10, ad.es.cluster.name=[appdynamics-events-service-cluster], ad.metric.processor.enabled.accounts=[], ad.accountmanager.keyNamesCSV=REDACTED, query.max.export.results.limit=65000, ad.es.response.filter.percentile.values.nullToZero=false, query.max.aggregation.level.limit.customers=10000, ad.metric.processor.enabled.configCreateAll=false, ad.es.event.index.isDocumentReplaceEnabled=false, ad.metric.processor.enabled.customEvents=false, query.max.scroll.batch.size=10000, ad.es.rolling.maxShardsPerIndex=25, query.inner.hits.size.limit=100, query.aggressive.search.call.timeout.millis=30000, ad.es.request.filter.partialPathsMap={"browserrecord": {"domreadytime": "metrics.domreadytime","enduserresponsetime": "metrics.enduserresponsetime","firstbytetime": "metrics.firstbytetime"},"mobilesessionrecord": {"durationMS": "metrics.durationMS"},"sessionrecord": {"durationMS": "metrics.durationMS","pagename": "browserRecords.pagename","pagetype": "browserRecords.pagetype"}}, ad.es.event.maxPublishRequestBodySizeBytes=1000000, ad.accountmanager.key.service=REDACTED, query.funnel.batch.scroll.size=10000, ad.es.event.index.maxFieldsPerIndex=3000, query.scroll.mode.enable=true, query.funnel.batch.concurrency.limit=3, ad.accountmanager.key.deprecatedKeys=REDACTED, ad.accountmanager.key.controller=REDACTED, ad.es.healthCheck.reservoir.expDecayFactor=0.015, ad.es.healthCheck.updateInterval=1 minutes, ad.metric.processor.disabled.accounts=[], query.default.scroll.expiry.millis=60000, ad.accountmanager.key.jf=REDACTED, ad.es.response.filter.boolean.eventTypesToFilter=[mobilesnapshot, mobilecrashreport, sessionrecord, mobilesessionrecord, synthsessionrecord], query.funnel.internal.results.limit=50000, query.max.top.level.aggregation.level.limit=1000, ad.es.healthCheck.reservoir.size=100, query.reject.time.unbounded.queries=false, ad.accountmanager.key.ops=REDACTED, ad.feature.adql.454.functions.envs={"default": "none", "dev": "all"}, ad.accountmanager.key.slm=REDACTED, query.cardinality.precision.threshold=-1, publish.validator.max.fields.per.event=255, ad.accountmanager.key.mds=REDACTED, ad.accountmanager.key.eum=REDACTED]   events-service-api-store / Connection to ElasticSearch: (unhealthy) java.net.ConnectException java.lang.RuntimeException: java.net.ConnectException at com.appdynamics.analytics.elasticsearch.client.ElasticSearchClientFacade.getNodesStatsResponse(ElasticSearchClientFacade.java:451) at com.appdynamics.analytics.processor.elasticsearch.clusters.ClusterHealthStateReporter.isPartOfCluster(ClusterHealthStateReporter.java:38) at com.appdynamics.analytics.processor.elasticsearch.clusters.AbstractClusterHealthStateReporter.updateHistogram(AbstractClusterHealthStateReporter.java:106) at com.appdynamics.analytics.processor.elasticsearch.clusters.AbstractClusterHealthStateReporter.updateHealthCheckResult(AbstractClusterHealthStateReporter.java:93) at com.appdynamics.common.util.health.AsynchronousHealthCheckable.runUpdateHealthCheckResult(AsynchronousHealthCheckable.java:123) at com.appdynamics.common.util.health.AsynchronousHealthCheckable$2.run(AsynchronousHealthCheckable.java:147) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.ConnectException at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:930) at org.elasticsearch.client.RestClient.performRequest(RestClient.java:300) at org.elasticsearch.client.RestClient.performRequest(RestClient.java:288) at com.appdynamics.analytics.elasticsearch.admin.ElasticSearchAdminFacade.getNodesStatsResponse(ElasticSearchAdminFacade.java:443) at com.appdynamics.analytics.elasticsearch.client.ElasticSearchClientFacade.getNodesStatsResponse(ElasticSearchClientFacade.java:449) ... 11 more Caused by: java.net.ConnectException at org.apache.http.nio.pool.RouteSpecificPool.timeout(RouteSpecificPool.java:168) at org.apache.http.nio.pool.AbstractNIOConnPool.requestTimeout(AbstractNIOConnPool.java:561) at org.apache.http.nio.pool.AbstractNIOConnPool$InternalSessionRequestCallback.timeout(AbstractNIOConnPool.java:822) at org.apache.http.impl.nio.reactor.SessionRequestImpl.timeout(SessionRequestImpl.java:183) at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processTimeouts(DefaultConnectingIOReactor.java:210) at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:155) at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:348) at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:191) at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64) ... 1 more     events-service-api-store / Connection to [http://0.0.0.0:9080/biz_outcome] with [DefaultBizOutcomeDefinitionClient]: (healthy)    events-service-api-store / Connection to [http://0.0.0.0:9080/events/query] with [DefaultAsyncQueryEventsClient]: (healthy)    events-service-api-store / Connection to [http://0.0.0.0:9080/events/query] with [DefaultQueryEventsClient]: (healthy)    events-service-api-store / Connection to [http://0.0.0.0:9080/v1/account] with [DefaultAccountServiceClient]: (healthy)    events-service-api-store / Connection to [http://0.0.0.0:9080/v1/accounts/meters] with [RestMeterServiceClient]: (healthy)    events-service-api-store / Connection to [http://0.0.0.0:9080/v1/admin/accounts] with [RestAccountsServiceClient]: (healthy)    events-service-api-store / Connection to [http://0.0.0.0:9080/v1/admin] with [AsyncJobScannerClient]: (healthy)    events-service-api-store / Connection to [http://0.0.0.0:9080/v1/admin] with [DefaultEventServiceAdminClient]: (healthy)    events-service-api-store / Connection to [http://0.0.0.0:9080/v1/jf] with [RestJobFrameworkClient]: (healthy)    events-service-api-store / Connection to [http://0.0.0.0:9080/v1/querymanagement/cancel] with [RestQueryManagementClient]: (healthy)    events-service-api-store / Connection to [http://0.0.0.0:9080/v1/slm] with [RestSlmPerfConfigsClient]: (healthy)    events-service-api-store / Connection to [http://0.0.0.0:9080/v3/accounts] with [RestEventTypeClient]: (healthy)    events-service-api-store / Connection to [http://0.0.0.0:9080/v3/events] with [DefaultEventServiceClient]: (healthy)    events-service-api-store / Resource [_ping - GET]: (healthy) Rates (Avg per second. Avg of last 5 min) success: [0.122272], user error: [0.000000], timeout: [0.000000], error: [0.000000]   events-service-api-store / Resource [entitysearch/sync - PUT]: (healthy) Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.074595]   events-service-api-store / Resource [v1/account - POST]: (healthy) Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.059934]   events-service-api-store / Resource [v1/elasticsearch/numberOfDataNodes - GET]: (healthy) Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.053746]   events-service-api-store / Resource [v1/store/report - GET]: (healthy) Rates (Avg per second. Avg of last 5 min) success: [0.074411], user error: [0.000000], timeout: [0.000000], error: [0.000000]   events-service-api-store / SystemAccessKeyAuthHandler: (healthy) deprecated: [], disabled: []   events-service-api-store / jobframework-module: (unhealthy) Job Framework instanceId: [SYS2.ZOOM.COM], number of jobs executed: [0], running since [<null>], currently executing jobs []   events-service-api-store / queues: (healthy) [1] queues [[biz-outcome-incoming-events] ratio: [0.00], size: [0], capacity: [1000]] ] [2023-12-03T12:11:47,688+03:00]  [INFO ]  [cache-refresh-scheduler-0]  [c.a.a.p.event.meter.DefaultMeters]  Quota refresh executor stats : [java.util.concurrent.ThreadPoolExecutor@29bd2b0f[Running, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]] [2023-12-03T12:11:47,688+03:00]  [INFO ]  [cache-refresh-scheduler-0]  [c.a.a.p.event.meter.DefaultMeters]  Submitted [0] tasks to refresh the quota remaining caches [2023-12-03T12:11:48,353+03:00]  [ERROR]  [main]  [c.a.a.p.e.n.s.ElasticsearchDependencyModule]  Elasticsearch was never healthy. com.github.rholder.retry.RetryException: Retrying failed to complete successfully after 40 attempts. at com.github.rholder.retry.Retryer.call(Retryer.java:174) at com.appdynamics.analytics.processor.elasticsearch.node.single.ElasticsearchDependencyModule.waitForHealthiness(ElasticsearchDependencyModule.java:96) at com.appdynamics.analytics.processor.elasticsearch.node.single.ElasticsearchDependencyModule.waitForHealthinessAndHandle(ElasticsearchDependencyModule.java:65) at com.appdynamics.analytics.processor.elasticsearch.node.single.ElasticsearchDependencyModule.lambda$registerDependencyMonitor$0(ElasticsearchDependencyModule.java:50) at io.dropwizard.lifecycle.setup.LifecycleEnvironment$ServerListener.lifeCycleStarted(LifecycleEnvironment.java:117) at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:194) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:74) at io.dropwizard.cli.ServerCommand.run(ServerCommand.java:53) at io.dropwizard.cli.EnvironmentCommand.run(EnvironmentCommand.java:45) at io.dropwizard.cli.ConfiguredCommand.run(ConfiguredCommand.java:87) at io.dropwizard.cli.Cli.run(Cli.java:78) at io.dropwizard.Application.run(Application.java:94) at com.appdynamics.common.framework.AbstractApp.callRunServer(AbstractApp.java:274) at com.appdynamics.common.framework.AbstractApp.runUsingFile(AbstractApp.java:268) at com.appdynamics.common.framework.AbstractApp.runUsingTemplate(AbstractApp.java:255) at com.appdynamics.common.framework.AbstractApp.runUsingTemplate(AbstractApp.java:175) at com.appdynamics.analytics.processor.AnalyticsService.main(AnalyticsService.java:76) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at com.appdynamics.common.executor.command.windows.AppRunServiceInternalCommand.execute(AppRunServiceInternalCommand.java:91) at com.appdynamics.common.executor.CommandExecutor.execute(CommandExecutor.java:38) at com.appdynamics.analytics.processor.executor.AnalyticsServiceExecutor.main(AnalyticsServiceExecutor.java:99) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at com.appdynamics.common.executor.standalone.ProxyMain.callActualMain(ProxyMain.java:165) at com.appdynamics.common.executor.standalone.ProxyMain.main(ProxyMain.java:106) Caused by: java.net.ConnectException: null at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:930) at org.elasticsearch.client.RestClient.performRequest(RestClient.java:300) at org.elasticsearch.client.RestClient.performRequest(RestClient.java:288) at co.elastic.clients.transport.rest_client.RestClientTransport.performRequest(RestClientTransport.java:147) at co.elastic.clients.elasticsearch.indices.ElasticsearchIndicesClient.exists(ElasticsearchIndicesClient.java:620) at co.elastic.clients.elasticsearch.indices.ElasticsearchIndicesClient.exists(ElasticsearchIndicesClient.java:636) at com.appdynamics.analytics.processor.util.startup.StartupHelpers.startupIndexExists(StartupHelpers.java:107) at com.appdynamics.analytics.processor.util.startup.StartupHelpers.upsertStartupIndex(StartupHelpers.java:97) at com.appdynamics.analytics.processor.util.startup.StartupHelpers.lambda$isElasticsearchHealthy$2(StartupHelpers.java:87) at com.appdynamics.common.util.locks.InterProcessClusterLock.acquireAndExecute(InterProcessClusterLock.java:35) at com.appdynamics.analytics.processor.util.startup.StartupHelpers.isElasticsearchHealthy(StartupHelpers.java:85) at com.appdynamics.analytics.processor.elasticsearch.node.single.ElasticsearchDependencyModule.lambda$waitForHealthiness$2(ElasticsearchDependencyModule.java:101) at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) at com.github.rholder.retry.Retryer.call(Retryer.java:160) ... 29 common frames omitted Caused by: java.net.ConnectException: null at org.apache.http.nio.pool.RouteSpecificPool.timeout(RouteSpecificPool.java:168) at org.apache.http.nio.pool.AbstractNIOConnPool.requestTimeout(AbstractNIOConnPool.java:561) at org.apache.http.nio.pool.AbstractNIOConnPool$InternalSessionRequestCallback.timeout(AbstractNIOConnPool.java:822) at org.apache.http.impl.nio.reactor.SessionRequestImpl.timeout(SessionRequestImpl.java:183) at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processTimeouts(DefaultConnectingIOReactor.java:210) at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:155) at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:348) at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:191) at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64) at java.base/java.lang.Thread.run(Unknown Source) [2023-12-03T12:11:48,354+03:00]  [ERROR]  [main]  [c.a.a.p.e.n.s.ElasticsearchDependencyModule]  Elasticsearch never successfully started up, stopping server startup. [2023-12-03T12:11:48,356+03:00]  [INFO ]  [main]  [c.a.common.framework.util.SimpleApp]  Stopping [events-service-api-store] [2023-12-03T12:11:48,384+03:00]  [INFO ]  [main]  [o.e.jetty.server.AbstractConnector]  Stopped application@4cf1ec2{HTTP/1.1, (http/1.1)}{0.0.0.0:9080} [2023-12-03T12:11:48,385+03:00]  [INFO ]  [main]  [o.e.jetty.server.AbstractConnector]  Stopped admin@16eb01d4{HTTP/1.1, (http/1.1)}{0.0.0.0:9081} [2023-12-03T12:11:48,385+03:00]  [INFO ]  [main]  [o.e.j.server.handler.ContextHandler]  Stopped i.d.j.MutableServletContextHandler@40bac624{/,null,STOPPED} [2023-12-03T12:11:48,385+03:00]  [INFO ]  [main]  [o.e.j.server.handler.ContextHandler]  Stopped i.d.j.MutableServletContextHandler@19740583{/,null,STOPPED} [2023-12-03T12:11:48,398+03:00]  [INFO ]  [main]  [c.a.a.p.c.m.s.DefaultMetricCorrelationService]  Stopped metric correlation service. [2023-12-03T12:11:48,398+03:00]  [INFO ]  [main]  [c.a.a.p.c.m.s.FileBasedCorrelationPersistor]  Stopped metric correlation file cleanup thread. [2023-12-03T12:11:48,399+03:00]  [INFO ]  [main]  [c.a.a.pipeline.framework.Pipelines]  Pipelines have stopped [2023-12-03T12:11:48,399+03:00]  [INFO ]  [main]  [c.a.a.p.j.JobFrameworkModule]  Stopping job framework scheduler [2023-12-03T12:11:48,399+03:00]  [INFO ]  [main]  [org.quartz.core.QuartzScheduler]  Scheduler QuartzScheduler_$_SYS2.ZOOM.COM shutting down. [2023-12-03T12:11:48,399+03:00]  [INFO ]  [main]  [org.quartz.core.QuartzScheduler]  Scheduler QuartzScheduler_$_SYS2.ZOOM.COM paused. [2023-12-03T12:11:48,400+03:00]  [INFO ]  [main]  [org.quartz.core.QuartzScheduler]  Scheduler QuartzScheduler_$_SYS2.ZOOM.COM shutdown complete. [2023-12-03T12:11:48,400+03:00]  [INFO ]  [main]  [c.a.a.p.j.JobFrameworkModule]  Stopped job framework scheduler [2023-12-03T12:11:48,400+03:00]  [INFO ]  [main]  [c.a.common.util.event.EventBuses]  Stopped [2023-12-03T12:11:49,415+03:00]  [WARN ]  [main]  [c.a.c.u.health.HealthReporterModule]  Task will be forcibly stopped now if it has not already stopped [2023-12-03T12:11:49,415+03:00]  [INFO ]  [main]  [c.a.c.u.health.HealthReporterModule]  Stopped [2023-12-03T12:11:49,423+03:00]  [INFO ]  [main]  [c.a.common.framework.util.SimpleApp]  Stopped [events-service-api-store] [2023-12-03T12:11:49,423+03:00]  [INFO ]  [main]  [c.a.a.p.e.n.s.ElasticsearchDependencyModule]  Successfully stopped all server components
Hello What is the best way to calculate sourcetypes size trend by time  index and level ?   i tried this two options but couldn't find a way to see the trend : index=_internal source=*license_usa... See more...
Hello What is the best way to calculate sourcetypes size trend by time  index and level ?   i tried this two options but couldn't find a way to see the trend : index=_internal source=*license_usage.log* type=Usage idx=*| eval GB=b/1024/1024/1024 | stats sum(GB) by st idx   index=* | eval raw_len=len(_raw)/1024/1024/1024 | stats sum(raw_len) as totalsize count as NumberOfEvent by index sourcetype | sort -NumberOfEvent | fields - NumberOfEvent
It's not exactly the same question. My question is how do I color a field according to another field The query looks like this: <query> stats count(Code) as count_by_id count(eval(like(message, ... See more...
It's not exactly the same question. My question is how do I color a field according to another field The query looks like this: <query> stats count(Code) as count_by_id count(eval(like(message, %READ-ERROR -&gt; DP is temporarily down%))) as read_error_count </query>   And I want to color the read_error_count column according to the count_by_id variable How do I do this in XML? <option name= "count">13</option> <option name= "dataOverlayMode">none</option> <option name= "drilldown">row</option> <option name= "wrap">true</option> <format type = "color" field= "read_error_count"> <colorPallete type="list"> <colorPalleteEntry></colorPalleteEntry> </colorPallete> </format>   Thank you @ITWhisperer 
FACT: ui-prefs is broken!
They did a real great mess, after 7.0, and some 8.x release Also with false in optimize_ui_prefs_performance, i'm now on 8.2.12 version, 1) optimize_ui_prefs_performance to true destroyes all ol... See more...
They did a real great mess, after 7.0, and some 8.x release Also with false in optimize_ui_prefs_performance, i'm now on 8.2.12 version, 1) optimize_ui_prefs_performance to true destroyes all old users customization on search tab ... also with optimize_ui_prefs_performance to false, 2) new ui-prefs.conf are not created anymore, only old ui-prefs are managed 3) also etc/users/launcher/local/ui-prefs.conf to remove "Explore Splunk Enterprise" banner has gone away! 4) users can't change Alerts/Reports/Dashboards object view modality (general/owner/app), since it's defaulted and reverted back to "All" next time you load the page!!! 5) seems ui-prefs is right managed only in "app/search/search|alerts|reports|dashboards" (default search App) This is really a great mess!!! Check here, https://community.splunk.com/t5/Splunk-Enterprise/quot-ui-prefs-conf-quot-no-more-working-from-Version-7-to/m-p/669361 We have had many users complain about this poor UI management!!!
Hi @GIA , good for you, see next time! let me know if I can help you more, or, please, accept one answer for the other people of Community. Ciao and happy splunking Giuseppe P.S.: Karma Points a... See more...
Hi @GIA , good for you, see next time! let me know if I can help you more, or, please, accept one answer for the other people of Community. Ciao and happy splunking Giuseppe P.S.: Karma Points are appreciated