Elastic Search Data Integrator - Malformed URL using special characters ?

[welo78]

Hello all,

I am trying to solve an issue with your addon for Splunk. Our client has an indice with name *:security-audit-*. Since we have a distributed environment we have heavy forwarders in cloud and indexers and searcheads in the on prem environment.

The URL is resolvable via nslookup and the endpoint is giving us a correct response if we try to connect to it via our username and password.

As you can see its transforming the : character into %3. Now, I am not sure if this is an actual issue, but our on-prem team is not receiving any data. I also tried using *security-audit-* which solved the errors but still no data has been recieved.

[elasticsearch_json://srvadm]
ca_certs_path = /opt/splunk/etc/auth/VWAG
date_field_name = @timestamp
elasticsearch_indice = *:security-audit-*
elasticsearch_instance_url = https://redacted:9243
greater_or_equal = {{ ansible_date_time.date }}
index = vw_de_aws_mlaas_apps
interval = 300
lower_or_equal = now
secret = {{ es_password }}
use_ssl = 1
user = siem_readonly
verify_certs = 0

root@fpea:/var/snap/amazon-ssm-agent/6312# cat /opt/splunk/var/log/splunk/ta_elasticsearch_data_integrator_modular_input_elasticsearch_json.log | grep security-audit

2022-10-21 11:52:38,943 WARNING pid=811606 tid=MainThread file=base.py:log_request_fail:299 | POST https://redacted:9243/*%3Asecurity-audit-*/_search?scroll=2m&size=1000 [status:403 request:0.032s]

2022-10-21 11:57:40,619 WARNING pid=814893 tid=MainThread file=base.py:log_request_fail:299 | POST https://redacted:9243/*%3Asecurity-audit-*/_search?scroll=2m&size=1000 [status:403 request:0.095s]

2022-10-21 12:02:41,357 WARNING pid=818040 tid=MainThread file=base.py:log_request_fail:299 | POST https://redacted:9243/*%3Asecurity-audit-*/_search?scroll=2m&size=1000 [status:403 request:0.091s]

2022-10-21 12:07:39,512 WARNING pid=820807 tid=MainThread file=base.py:log_request_fail:299 | POST https://redacted:9243/*%3Asecurity-audit-*/_search?scroll=2m&size=1000 [status:403 request:0.066s]

2022-10-21 12:12:46,422 WARNING pid=823706 tid=MainThread file=base.py:log_request_fail:299 | POST https://redacted:9243/*%3Asecurity-audit-*/_search?scroll=2m&size=1000 [status:403 request:0.036s]

[welo78]

I contacted the developer of the integratior himself and this is his reponse. I hope anybody finds this helpful.

The issue here (as in the official Elastic documentation) is that the use of a colon (:) has been deprecated since version 7.0+:  

See doc: https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html#indices-cr...

and the Elasticsearch Integrator is also using 7.0+ python libraries.