All Apps and Add-ons

Elastic Search Data Integrator - Malformed URL using special characters ?

welo78
Explorer

Hello all,

I am trying to solve an issue with your addon for Splunk. Our client has an indice with name *:security-audit-*. Since we have a distributed environment we have heavy forwarders in cloud and indexers and searcheads in the on prem environment.

The URL is resolvable via nslookup and the endpoint is giving us a correct response if we try to connect to it via our username and password.

As you can see its transforming the : character into %3. Now, I am not sure if this is an actual issue, but our on-prem team is not receiving any data. I also tried using *security-audit-* which solved the errors but still no data has been recieved.

[elasticsearch_json://srvadm]
ca_certs_path = /opt/splunk/etc/auth/VWAG
date_field_name = @timestamp
elasticsearch_indice = *:security-audit-*
elasticsearch_instance_url = https://redacted:9243
greater_or_equal = {{ ansible_date_time.date }}
index = vw_de_aws_mlaas_apps
interval = 300
lower_or_equal = now
secret = {{ es_password }}
use_ssl = 1
user = siem_readonly
verify_certs = 0

 

root@fpea:/var/snap/amazon-ssm-agent/6312# cat /opt/splunk/var/log/splunk/ta_elasticsearch_data_integrator_modular_input_elasticsearch_json.log | grep security-audit

2022-10-21 11:52:38,943 WARNING pid=811606 tid=MainThread file=base.py:log_request_fail:299 | POST https://redacted:9243/*%3Asecurity-audit-*/_search?scroll=2m&size=1000 [status:403 request:0.032s]

2022-10-21 11:57:40,619 WARNING pid=814893 tid=MainThread file=base.py:log_request_fail:299 | POST https://redacted:9243/*%3Asecurity-audit-*/_search?scroll=2m&size=1000 [status:403 request:0.095s]

2022-10-21 12:02:41,357 WARNING pid=818040 tid=MainThread file=base.py:log_request_fail:299 | POST https://redacted:9243/*%3Asecurity-audit-*/_search?scroll=2m&size=1000 [status:403 request:0.091s]

2022-10-21 12:07:39,512 WARNING pid=820807 tid=MainThread file=base.py:log_request_fail:299 | POST https://redacted:9243/*%3Asecurity-audit-*/_search?scroll=2m&size=1000 [status:403 request:0.066s]

2022-10-21 12:12:46,422 WARNING pid=823706 tid=MainThread file=base.py:log_request_fail:299 | POST https://redacted:9243/*%3Asecurity-audit-*/_search?scroll=2m&size=1000 [status:403 request:0.036s]

 

Labels (2)
0 Karma

welo78
Explorer

I contacted the developer of the integratior himself and this is his reponse. I hope anybody finds this helpful.

The issue here (as in the official Elastic documentation) is that the use of a colon (:) has been deprecated since version 7.0+:  

See doc: https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html#indices-cr...

and the Elasticsearch Integrator is also using 7.0+ python libraries.

Get Updates on the Splunk Community!

Streamline Data Ingestion With Deployment Server Essentials

REGISTER NOW!Every day the list of sources Admins are responsible for gets bigger and bigger, often making the ...

Remediate Threats Faster and Simplify Investigations With Splunk Enterprise Security ...

REGISTER NOW!Join us for a Tech Talk around our latest release of Splunk Enterprise Security 7.2! We’ll walk ...

Introduction to Splunk AI

WATCH NOWHow are you using AI in Splunk? Whether you see AI as a threat or opportunity, AI is here to stay. ...