Hello Team,
Can anyone please help me out to clarify the following query and a better approach for deploying the Observability solution?
I have an Application which is deployed as High Availability Solution, as in it acts as Primary/Secondary, so the application runs on either of the node at a time.
Now we are integrating our application with Splunk Enterprise for Observability. As part of the solution, we are deploying Splunk Otel Collector + FluentD agent
to collect the metrics/logs/traces.
Now how do we manage the integration solution, as in if the Application is running on HOST A, I need to make sure both these agents (Splunk Otel Collector + FluentD) to be up
and running on HOST A to collect & ingest data into Splunk Enterprise, and the agents on the other HOST B, needs to be IDLE so that we don't ingest data into Splunk.
This can be achieved my deploying custom script (to be executed under Cron frequently say 5 mins to check where the Application is Active and start the agent services accordingly).
But how do we make sure the data that are ingested into Splunk are appropriate (without any duplicates) when handling this scenario because there are 2 different hosts?
We also would like to avoid a drop down in the Dashboard to select appropriate HOST to filter the data based on the HOST? Because this procedure makes hard for the business team to
understand where the application is running currently and select the HOST accordingly? so this approach does not make great sense to me.
Is there a better approach to handle this situation? In case if we are having Load Balancer for the Application, Are we able to make use of it to tell Splunk otel collector + Fluentd
to collect data only from active Host and then send the data through HTTP Event Collector.
