Splunk dashboard "could not create search" on external network

[jadengoho]

Hi All,
I have a Splunk environment which works internally using the IP address.
But when I tried accessing it externally on a site -sometimes dashboards are working and sometimes it's not showing anything just "Could not create search".
Is this caused by a network issue or Splunk configuration file?
Jobs are randomly stopped and show "502 error" on network inspect element.

[esix_splunk]

Most likely this is related to either a load balancer or application firewall/filter you are going through when you access this externally. Generally we recommend that in the case of load balancers, that you make sure sticky sessions are enabled or that you disable application filtering for the Splunk base urls.

[jadengoho]

Hi, esix,
What do you mean by "sticky sessions are enabled".
And how can I disable application filtering for Splunk base URL?

[gcusello]

Hi jadengoho,
some very stupid questions:

• when you say "sometimes", are you saying that the same dashboard, sometimes run and sometimes goes in error or that any dashboards (always the same) go in error and any always run?
• did you verified in your dashboard that all the configurations use IP address and not hostnames?

Bye.
Giuseppe

[jadengoho]

when you say "sometimes", are you saying that the same dashboard, sometimes runs and sometimes goes in error or that any dashboards (always the same) go in error and any always run? - sometimes panels works, sometimes not.. It always happens

did you verify in your dashboard that all the configurations use IP address and not hostnames? Yes we tried using IP address same issue

[gcusello]

Hi jadengoho,
did you find the differences between the two kind of panels?
Bye.
Giuseppe

[jadengoho]

External Site: All panels show this error.
Internal Site: No error Exist.

We suspect that the error occurs on the load balancer or the firewall.

[gcusello]

Hi jadengoho,
are there firewalls between splunk servers?
anyway you can test the open ports between servers.

Why do you use a load balancer between Search Heads and Indexers? you don't need them: Splunk has auto load balancing features.

Bye.
Giuseppe

[jawaharas]

Should be a network issue. Is it happening even if you try from different network (say your mobile network instead of WiFi) ?

[jadengoho]

Same network, but using the external address still shows the issue.