Splunk Search

What are the consequences of disabling real-time searches or converting them to saved searches for a Prod environment?

gagandeep_arora
Path Finder

Hello All,

We are in the process of cleaning up unused and Real Time Searches from the system.
I can see there are two real time searches run from Distributed Management Console only on my Deployment Manager:
http_event_collector_instance
http_event_collector_deployment

HTTP Event Collector: Instance ------- (when I load this page, It only gives me below output).
You currently have no tokens configured.

Search query used in this dashboard:

dmc_set_index_introspection` component="HttpEventCollector" data.series="$data_series$" host="$host$" $token_clause$
    | bin _time span=1m
    | stats sum(data.num_of_events) as events_total, sum(data.num_of_requests) as requests_total, sum(data.num_of_requests_to_disabled_token) as disabled_token_total, sum(data.num_of_requests_to_incorrect_url) as incorrect_url_total, sum(data.num_of_auth_failures) as auth_fail_total, sum(data.num_of_parser_errors) as parser_error_total, sum(data.total_bytes_indexed) as data_indexed, sum(data.total_bytes_received) as data_received by _time
    | eval incorrect_url_total=if(isnotnull(incorrect_url_total), incorrect_url_total, 0)
    | eval auth_fail_total=if(isnotnull(auth_fail_total), auth_fail_total, 0)
    | eval data_indexed=data_indexed/pow(1024, 2)
    | eval data_received=data_received/pow(1024, 2)
    | eval valid_requests_total = requests_total
    | eval invalid_requests_total = auth_fail_total + disabled_token_total + incorrect_url_total

I need help in clarifying below points before disabling the searches:
1. What does this search do?
2. What will be the impact if I disable this or convert it into a saved search in place of real time search?
3. How to make sure that it is not further referred in other dashboards as it only belongs to DMC (Distributed Management Console)?

0 Karma

woodcock
Esteemed Legend

Um.... What makes you think that this is a real-time search? Even if it is, you do realize that it will only be running when the dashboard is open, right? And it appears that you are not using HEC (Http Event Collector) so there is no reason for anybody to be opening that dashboard. Furthermore, if this is your Deployer OR your Monitoring Console, there is no reason for ANYBODY other than YOU, the main admin, to ever be there in the first place. Do you really need to lock out yourself from doing things that only you can do but that you have utterly no reason to ever do? In other words, just exactly what is the problem that you think that you need to solve here?

gagandeep_arora
Path Finder

Thanks Both for your prompt responses. We were seeing performance issue in the environment, Critical alerts were getting skipped due to long queues of searches. We were in the process of cleaning the unused searches and scheduled searches running every 1 min and cleaning of real time searches. We also followed the same approach of disabling the complete app if no one is using or taking consent from the user. We seems to be good now in this regard.

DalJeanis
SplunkTrust
SplunkTrust

I've GOT to second @woodcock's opinion, based on the fact that you are thinking of "cleaning up" things that are integral parts of apps that you have deployed but not used.

If you want to get rid of the app, get rid of the app, don't just hack parts of it off because they haven't been used recently.

If you are trying to get rid of real time searches or convert them to periodic searches... which can be a good architectural decision... then start with the ones that have actually run in the last three months.

0 Karma