I have a Splunk environment which works internally using the IP address.
But when I tried accessing it externally on a site -sometimes dashboards are working and sometimes it's not showing anything just "Could not create search".
Is this caused by a network issue or Splunk configuration file?
Jobs are randomly stopped and show "502 error" on network inspect element.
Should be a network issue. Is it happening even if you try from different network (say your mobile network instead of WiFi) ?
some very stupid questions:
when you say "sometimes", are you saying that the same dashboard, sometimes runs and sometimes goes in error or that any dashboards (always the same) go in error and any always run? - sometimes panels works, sometimes not.. It always happens
did you verify in your dashboard that all the configurations use IP address and not hostnames? Yes we tried using IP address same issue
External Site: All panels show this error.
Internal Site: No error Exist.
We suspect that the error occurs on the load balancer or the firewall.
are there firewalls between splunk servers?
anyway you can test the open ports between servers.
Why do you use a load balancer between Search Heads and Indexers? you don't need them: Splunk has auto load balancing features.
Most likely this is related to either a load balancer or application firewall/filter you are going through when you access this externally. Generally we recommend that in the case of load balancers, that you make sure sticky sessions are enabled or that you disable application filtering for the Splunk base urls.
What do you mean by "sticky sessions are enabled".
And how can I disable application filtering for Splunk base URL?