topic Re: Starting Events Service cluster in Splunk AppDynamics

Starting Events Service cluster

Abdulrahman_Ka1 — Tue, 01 Jun 2021 08:59:19 GMT

I'm trying to start the Events service nodes but it doesn't work

( 3/ 7) Start Events Service Cluster: Starting the Events Service api store node
( 3/ 7) Start Events Service Cluster: FAILED
Job failed to execute.
Failure occurred: Start Events Service Cluster
Error message:
Task failed: Starting the Events Service api store node
on host: 10.0.30.45
as user: root
with message: Connection to [http://10.0.30.45:9080/_ping] failed due to [Failed to connect to /10.0.30.45:9080].

[root@console platform-admin]# bin/platform-admin.sh submit-job --platform-name AppDPlatform --service events-service --job start( 1/ 7) Load Events Service cluster configuration: SUCCESS( 2/ 7) Load all ES cluster hosts into execution context.: SUCCESS ( 3/ 7) Start Events Service Cluster: Starting the Events Service api store node( 3/ 7) Start Events Service Cluster: FAILEDJob failed to execute.Failure occurred: Start Events Service ClusterError message:Task failed: Starting the Events Service api store nodeon host: 10.0.30.45as user: rootwith message: Connection to [http://10.0.30.45:9080/_ping] failed due to [Failed to connect to /10.0.30.45:9080].

I also tried to start each node alone I get this error

Spoiler

|-Xmx2990m|-Xms2990m|-Djava.net.preferIPv4Stack=true|-Dfile.encoding=UTF-8|-Djute.maxbuffer=30000000|-Dzookeeper.snapshot.trust.empty=true|-XX:+UseParNewGC|-XX:+UseConcMarkSweepGC|-XX:CMSInitiatingOccupancyFraction=75|-XX:+UseCMSInitiatingOccupancyOnly|-XX:+DisableExplicitGC|-XX:+PrintGCDetails|-XX:+PrintGCDateStamps|-XX:+PrintGCTimeStamps|-XX:+PrintClassHistogram|-XX:+PrintTenuringDistribution|-XX:+PrintGCApplicationStoppedTime|-XX:+PrintPromotionFailure|-verbose:gc|-XX:GCLogFileSize=64m|-XX:NumberOfGCLogFiles=4|-XX:+UseGCLogFileRotation|-XX:CompileCommand=exclude,org/apache/lucene/lucene54/Lucene54DocValuesConsumer.addSortedNumericField|-XX:CompileCommand=exclude,org/apache/lucene/lucene54/Lucene54DocValuesConsumer.addBinaryField|-XX:CompileCommand=exclude,org/elasticsearch/search/aggregations/metrics/percentiles/tdigest/AbstractTDigestPercentilesAggregator$1.collect|-XX:CompileCommand=exclude,org/apache/lucene/index/SortedNumericDocValuesWriter.flush|-XX:CompileCommand=exclude,org/apache/lucene/codecs/PushPostingsWriterBase.writeTerm|-Xloggc:"/opt/appdynamics/platform/product/events-service/processor/bin/../logs/%p-gc.log"|-DAPPLICATION_HOME="/opt/appdynamics/platform/product/events-service/processor/bin/.."|-classpath|"/opt/appdynamics/platform/product/events-service/processor/bin/../lib/*"|com.appdynamics.analytics.processor.AnalyticsService|-p|"/opt/appdynamics/platform/product/events-service/processor/./conf/events-service-api-store.properties"|-y|"/opt/appdynamics/platform/product/events-service/processor/bin/../conf/events-service-api-store.yml"

CompilerOracle: exclude org/apache/lucene/lucene54/Lucene54DocValuesConsumer.addSortedNumericField
CompilerOracle: exclude org/apache/lucene/lucene54/Lucene54DocValuesConsumer.addBinaryField
CompilerOracle: exclude org/elasticsearch/search/aggregations/metrics/percentiles/tdigest/AbstractTDigestPercentilesAggregatorstart.collect
CompilerOracle: exclude org/apache/lucene/index/SortedNumericDocValuesWriter.flush
CompilerOracle: exclude org/apache/lucene/codecs/PushPostingsWriterBase.writeTerm
10:52:52.934 [main] INFO com.appdynamics.analytics.processor.AnalyticsService - Starting analytics processor with arguments [-p, /opt/appdynamics/platform/product/events-service/processor/./conf/events-service-api-store.properties, -y, /opt/appdynamics/platform/product/events-service/processor/bin/../conf/events-service-api-store.yml]
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console. Set system property 'log4j2.debug' to show Log4j2 internal initialization logging.

pwd/opt/appdynamics/platform/product/events-service/processor[root@events-service-03 processor]# echo $JAVA_HOME/usr/lib/jvm/jre-1.8.0-openjdk-1.8.0.292.b10-1.el7_9.x86_64/[root@events-service-03 processor]# ./bin/events-service.sh start -p ./conf/events-service-api-store.properties &[1] 43742[root@events-service-03 processor]# ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console. Set system property 'log4j2.debug' to show Log4j2 internal initialization logging.[2021-06-01T10:52:52,627+02:00] [APPLICATION_HOME] has been overridden to [/opt/appdynamics/platform/product/events-service/processor/bin/..][2021-06-01T10:52:52,631+02:00] [APPLICATION_HOME] has been overridden to [/opt/appdynamics/platform/product/events-service/processor/bin/..][2021-06-01T10:52:52,636+02:00] Program arguments: |-Xmx2990m|-Xms2990m|-Djava.net.preferIPv4Stack=true|-Dfile.encoding=UTF-8|-Djute.maxbuffer=30000000|-Dzookeeper.snapshot.trust.empty=true|-XX:+UseParNewGC|-XX:+UseConcMarkSweepGC|-XX:CMSInitiatingOccupancyFraction=75|-XX:+UseCMSInitiatingOccupancyOnly|-XX:+DisableExplicitGC|-XX:+PrintGCDetails|-XX:+PrintGCDateStamps|-XX:+PrintGCTimeStamps|-XX:+PrintClassHistogram|-XX:+PrintTenuringDistribution|-XX:+PrintGCApplicationStoppedTime|-XX:+PrintPromotionFailure|-verbose:gc|-XX:GCLogFileSize=64m|-XX:NumberOfGCLogFiles=4|-XX:+UseGCLogFileRotation|-XX:CompileCommand=exclude,org/apache/lucene/lucene54/Lucene54DocValuesConsumer.addSortedNumericField|-XX:CompileCommand=exclude,org/apache/lucene/lucene54/Lucene54DocValuesConsumer.addBinaryField|-XX:CompileCommand=exclude,org/elasticsearch/search/aggregations/metrics/percentiles/tdigest/AbstractTDigestPercentilesAggregator$1.collect|-XX:CompileCommand=exclude,org/apache/lucene/index/SortedNumericDocValuesWriter.flush|-XX:CompileCommand=exclude,org/apache/lucene/codecs/PushPostingsWriterBase.writeTerm|-Xloggc:"/opt/appdynamics/platform/product/events-service/processor/bin/../logs/%p-gc.log"|-DAPPLICATION_HOME="/opt/appdynamics/platform/product/events-service/processor/bin/.."|-classpath|"/opt/appdynamics/platform/product/events-service/processor/bin/../lib/*"|com.appdynamics.analytics.processor.AnalyticsService|-p|"/opt/appdynamics/platform/product/events-service/processor/./conf/events-service-api-store.properties"|-y|"/opt/appdynamics/platform/product/events-service/processor/bin/../conf/events-service-api-store.yml" CompilerOracle: exclude org/apache/lucene/lucene54/Lucene54DocValuesConsumer.addSortedNumericFieldCompilerOracle: exclude org/apache/lucene/lucene54/Lucene54DocValuesConsumer.addBinaryFieldCompilerOracle: exclude org/elasticsearch/search/aggregations/metrics/percentiles/tdigest/AbstractTDigestPercentilesAggregatorstart.collectCompilerOracle: exclude org/apache/lucene/index/SortedNumericDocValuesWriter.flushCompilerOracle: exclude org/apache/lucene/codecs/PushPostingsWriterBase.writeTerm10:52:52.934 [main] INFO com.appdynamics.analytics.processor.AnalyticsService - Starting analytics processor with arguments [-p, /opt/appdynamics/platform/product/events-service/processor/./conf/events-service-api-store.properties, -y, /opt/appdynamics/platform/product/events-service/processor/bin/../conf/events-service-api-store.yml]ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console. Set system property 'log4j2.debug' to show Log4j2 internal initialization logging.

Re: Starting Events Service cluster

iamryan — Tue, 01 Jun 2021 18:20:54 GMT

Hi @Abdulrahman.Kazamel,

I couldn't find any relevant info on those errors, but I did find this AppD Knowledge base article that describes how to turn on debugging mode. Perhaps this will give you more insight into the issue.

https://community.appdynamics.com/t5/Knowledge-Base/How-do-I-enable-the-Events-Service-debugging-port/ta-p/26128

Re: Starting Events Service cluster

Abdulrahman_Ka1 — Wed, 02 Jun 2021 16:37:23 GMT

Hi @Ryan.Paredez

I made some edit like

ssh by hostname, not only with the IP address

maximize each node to 25 G ram

reinstalled the events service cluster by

bin/platform-admin.sh install-events-service --profile prod --hosts 10.0.30.45 10.0.30.46 10.0.30.47 --data-dir /opt/appdynamics/eventsservice --platform-name AppDPlatform

also in each machine

iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 9080 -j ACCEPT
iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 9081 -j ACCEPT
iptables -A INPUT -m state --state NEW -m multiport -p tcp --dports 9300:9400 -j ACCEPT

however, the firewall was off

and tried again to restarted the cluster by

Spoiler

[root@console platform-admin]# bin/platform-admin.sh submit-job --platform-name AppDPlatform --service events-service --job restart-cluster
( 1/ 6) Load Events Service cluster configuration: SUCCESS
( 2/ 6) Load all ES cluster hosts into execution context.: SUCCESS
( 3/ 6) Disable transient allocations on masters: Disable transient allocation in the api store node
( 3/ 6) Disable transient allocations on masters: SUCCESS
( 4/ 6) Restart all nodes: Stop the Events Service api store node
( 4/ 6) Restart all nodes: Starting the Events Service api store node
( 4/ 6) Restart all nodes: Stop the Events Service api store node
( 4/ 6) Restart all nodes: Starting the Events Service api store node
( 4/ 6) Restart all nodes: Stop the Events Service api store node
( 4/ 6) Restart all nodes: Starting the Events Service api store node
( 4/ 6) Restart all nodes: SUCCESS
( 5/ 6) Enable transient allocations on masters: Enable transient allocation in the api store node
( 5/ 6) Enable transient allocations on masters: FAILED
Job failed to execute.
Failure occurred: Enable transient allocations on masters
Error message:
Task failed: Enable transient allocation in the api store node
on host: 10.0.30.45
as user: root
with message: Connection to [http://10.0.30.45:9080/v1/elasticsearch/_cluster/settings] failed due to [Failed to connect to /10.0.30.45:9080].

[root@console platform-admin]# bin/platform-admin.sh submit-job --platform-name AppDPlatform --service events-service --job restart-cluster( 1/ 6) Load Events Service cluster configuration: SUCCESS( 2/ 6) Load all ES cluster hosts into execution context.: SUCCESS( 3/ 6) Disable transient allocations on masters: Disable transient allocation in the api store node( 3/ 6) Disable transient allocations on masters: SUCCESS( 4/ 6) Restart all nodes: Stop the Events Service api store node( 4/ 6) Restart all nodes: Starting the Events Service api store node( 4/ 6) Restart all nodes: Stop the Events Service api store node( 4/ 6) Restart all nodes: Starting the Events Service api store node( 4/ 6) Restart all nodes: Stop the Events Service api store node( 4/ 6) Restart all nodes: Starting the Events Service api store node( 4/ 6) Restart all nodes: SUCCESS( 5/ 6) Enable transient allocations on masters: Enable transient allocation in the api store node( 5/ 6) Enable transient allocations on masters: FAILEDJob failed to execute.Failure occurred: Enable transient allocations on mastersError message:Task failed: Enable transient allocation in the api store nodeon host: 10.0.30.45as user: rootwith message: Connection to [http://10.0.30.45:9080/v1/elasticsearch/_cluster/settings] failed due to [Failed to connect to /10.0.30.45:9080].

Re: Starting Events Service cluster

iamryan — Thu, 03 Jun 2021 17:31:36 GMT

Hi @Abdulrahman.Kazamel,

Thanks for the additional info. Let's see if the Community can jump in and help you out. In the meantime, if you happen to find a solution, please do come back and share the results here. Knowledge sharing is key to the peer-to-peer community.

Re: Starting Events Service cluster

Abdulrahman_Ka1 — Sun, 27 Jun 2021 15:20:24 GMT

Hello @Ryan.Paredez

I made a couple of configuration and at the end

each time I do the installation the ports in each machine changes to 0.0.0.0

For now, the issue was in DNS resolve as I had to make it manual in each machine at
/etc/hosts file
to give an example

eventsservice-01 10.0.30.xx
eventsservice-02 10.0.30.xx
eventsservice-03 10.0.30.xx

the installation done successfully after this configuration and yet still checking the health as it's not good.

*another update *

after the last configuration, the events service were not able to keep running so,

I changed every hostname to IP in the conf/events-service-api-store.properties in each machine.
I stopped/disabled the firewall and iptables,
and now it's up and running in green health since two hours!

kindly I would like to thank you for your support.

Thanks
Abdulrahman

Re: Starting Events Service cluster

Aribowo_Soegiar — Wed, 07 Jun 2023 07:19:14 GMT

Hi @Abdulrahman.Kazamel,

I also encountered the similar issue as follows:

I want to verify, what you did to solve the issue are the following:

- add the event service machine to /etc/hosts file

- changed every hostname to IP in the conf/events-service-api-store.properties

Am I correct? Or is there any other changes need to be done.

Thank you.

Aribowo

Re: Starting Events Service cluster

iamryan — Fri, 09 Jun 2023 16:59:57 GMT

Hi @Aribowo.Soegiarto,

Since this post is about 2 years old, you may want to reask the question on the forums, or you can reach out to your AppD Rep, or even contact AppD Support.

How do I submit a Support ticket? An FAQ

Re: Starting Events Service cluster

Luiz_Polli1 — Tue, 20 Jun 2023 13:49:28 GMT

Hi Aribowo,

Same issue here.

Thanks,

Luiz Polli

Re: Starting Events Service cluster

iamryan — Tue, 20 Jun 2023 16:06:22 GMT

Hi @Luiz.Polli,

As I mentioned to @Aribowo.Soegiarto, given how old this post is, I think contacting AppD support is best. I was not able to find a ticket created by Aribowo.How do I submit a Support ticket? An FAQ

Re: Starting Events Service cluster

Luiz_Polli1 — Tue, 20 Jun 2023 20:35:36 GMT

Hi Ryan,
Creating a ticket now.

Thank you,
Luiz Polli

Re: Starting Events Service cluster

iamryan — Tue, 27 Jun 2023 16:45:26 GMT

Hi @Luiz.Polli,

Thanks for creating the ticket, if you can share the results/solution once the ticket has reached closure, that would be awesome.

Re: Starting Events Service cluster

Luiz_Polli1 — Tue, 27 Jun 2023 17:35:08 GMT

Hi @Ryan.Paredez,
Of course. Let you all know.
Thank you