All Apps and Add-ons

How to configure the Elasticsearch Data INtegrator

robertlynch2020
Motivator

Hi

In our company some logs are put to Elasticsearch, i have downloaded the Elasticsearch Data INtegrator to see if i can pull out the data.

When i log into Grafana i can see the configuration on how it access Elasticsearch

alt text

However when i try to reproduce it, i get errors.
alt text

ta_elasticsearch_data_integrator_modular_input_elasticsearch_json.log
2019-10-31 12:17:20,493 INFO pid=29104 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1
2019-10-31 12:17:22,408 INFO pid=29104 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1
2019-10-31 12:17:26,011 INFO pid=29104 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1
2019-10-31 12:17:29,643 ERROR pid=29104 tid=MainThread file=base_modinput.py:log_error:307 | Get error when collecting events.
Traceback (most recent call last):
File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/modinput_wrapper/base_modinput.py", line 127, in stream_events
self.collect_events(ew)
File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/elasticsearch_json.py", line 104, in collect_events
input_module.collect_events(self, ew)
File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/input_module_elasticsearch_json.py", line 49, in collect_events
opt_ca_certs_path = opt_ca_certs_path.strip()
AttributeError: 'NoneType' object has no attribute 'strip'

OR
splunks.log

10-31-2019 12:15:00.828 +0100 ERROR ExecProcessor - message from "python /hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/elasticsearch_json.py" Traceback (most recent call last):

I am new to Elasticsearch, so perhaps i am missing something easy? Any help would be great thanks

Rob

0 Karma
1 Solution

gaurav_maniar
Builder

Hi @robertlynch2020 ,

The issue is related to CA Certificate path variable in the application script.
I don't have exact solution to your problem, but we can do trial & error check.

First, this is python variable value related error.
Now open python file - /hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/input_module_elasticsearch_json.py
and comment the line number 49 (put '#' at start of the line to convert it to comment)

Once above changes are done check the logs and let me know what the new errors you are getting.
(no need to restart splunk for python scripts related changes)

View solution in original post

0 Karma

gaurav_maniar
Builder

Hi @robertlynch2020 ,

The issue is related to CA Certificate path variable in the application script.
I don't have exact solution to your problem, but we can do trial & error check.

First, this is python variable value related error.
Now open python file - /hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/input_module_elasticsearch_json.py
and comment the line number 49 (put '#' at start of the line to convert it to comment)

Once above changes are done check the logs and let me know what the new errors you are getting.
(no need to restart splunk for python scripts related changes)

0 Karma

robertlynch2020
Motivator

cheers for help @gaurav_maniar

FYI This is my current configuration. I have made the changes you said and i disablede and enabled the connection. I am getting the following now.

 #opt_ca_certs_path = opt_ca_certs_path.strip()

[elasticsearch_json://esearch]
date_field_name = body.ts
elasticsearch_indice = metric-2019.11.01
elasticsearch_instance_url = http://mx12405vm
greater_or_equal = 2019-01-01
index = mlc_test
interval = 60
lower_or_equal = now
port = 10212
use_ssl = False
verify_certs = False
user =
secret =
sourcetype = elasticsearch


2019-11-01 13:17:01,163 INFO pid=13710 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1
2019-11-01 13:17:03,063 INFO pid=13710 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1
2019-11-01 13:17:06,505 INFO pid=13710 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1
2019-11-01 13:17:09,953 INFO pid=13710 tid=MainThread file=setup_util.py:log_info:114 | Log level is not set, use default INFO
2019-11-01 13:17:09,954 WARNING pid=13710 tid=MainThread file=base.py:log_request_fail:146 | GET http://mx12405vm:10212/metric-2019.11.01/_search?scroll=2m&size=1000 [status:N/A request:0.000s]
Traceback (most recent call last):
  File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/connection/http_urllib3.py", line 232, in perform_request
    method, url, body, retries=Retry(False), headers=request_headers, **kw
  File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/urllib3/util/retry.py", line 343, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/urllib3/connectionpool.py", line 354, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/hp737srv2/apps/splunk/lib/python2.7/httplib.py", line 1042, in request
    self._send_request(method, url, body, headers)
  File "/hp737srv2/apps/splunk/lib/python2.7/httplib.py", line 1082, in _send_request
    self.endheaders(body)
  File "/hp737srv2/apps/splunk/lib/python2.7/httplib.py", line 1038, in endheaders
    self._send_output(message_body)
  File "/hp737srv2/apps/splunk/lib/python2.7/httplib.py", line 882, in _send_output
    self.send(msg)
  File "/hp737srv2/apps/splunk/lib/python2.7/httplib.py", line 844, in send
    self.connect()
  File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/urllib3/connection.py", line 196, in connect
    conn = self._new_conn()
  File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/urllib3/connection.py", line 180, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fce6168fd10>: Failed to establish a new connection: getaddrinfo() argument 2 must be integer or string
2019-11-01 13:17:10,955 WARNING pid=13710 tid=MainThread file=base.py:log_request_fail:146 | GET http://mx12405vm:10212/metric-2019.11.01/_search?scroll=2m&size=1000 [status:N/A request:0.000s]
Traceback (most recent call last):
  File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/connection/http_urllib3.py", line 232, in perform_request
    method, url, body, retries=Retry(False), headers=request_headers, **kw
  File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/urllib3/util/retry.py", line 343, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/urllib3/connectionpool.py", line 354, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/hp737srv2/apps/splunk/lib/python2.7/httplib.py", line 1042, in request
    self._send_request(method, url, body, headers)
  File "/hp737srv2/apps/splunk/lib/python2.7/httplib.py", line 1082, in _send_request
    self.endheaders(body)
  File "/hp737srv2/apps/splunk/lib/python2.7/httplib.py", line 1038, in endheaders
    self._send_output(message_body)
  File "/hp737srv2/apps/splunk/lib/python2.7/httplib.py", line 882, in _send_output
    self.send(msg)
  File "/hp737srv2/apps/splunk/lib/python2.7/httplib.py", line 844, in send
    self.connect()
  File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/urllib3/connection.py", line 196, in connect
    conn = self._new_conn()
  File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/urllib3/connection.py", line 180, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fce6168ff10>: Failed to establish a new connection: getaddrinfo() argument 2 must be integer or string
^C
0 Karma

gaurav_maniar
Builder

Again python related error.

now open python file - /hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/urllib3/connection.py

Goto line number 171,

(self._dns_host, self.port), self.timeout, **extra_kw)

change it to following, changing port from unicode to string,

(self._dns_host, self.port.encode('ascii')), self.timeout, **extra_kw)

Previously it worked on windows, give it a try on linux.

0 Karma

robertlynch2020
Motivator

Hi @gaurav_maniar

It now works now that i update the date_field_name = timestamp.
Thanks so so much 🙂

[elasticsearch_json://esearch]
date_field_name = timestamp
elasticsearch_indice = metric-2019.11.01
elasticsearch_instance_url = http://mx12405vm
greater_or_equal = 2019-01-01
index = mlc_test
interval = 60
lower_or_equal = now
port = 10212
use_ssl = False
verify_certs = False
user =
secret =
sourcetype = elasticsearch

0 Karma

Chika
Loves-to-Learn Lots

The Elasticsearch Data Integrator appears to be not responsive when a non-admin user tries to access the inputs and configuration menu. Any help why this is happening?

0 Karma

robertlynch2020
Motivator

Just to add i am still gettting errors but the basic connection is thre.

OVI4Q2JpWGdWR2ZOYjNBAAAAAAAANn8Wb1B6ZmszUzlSOENiaVhnVkdmTmIzQQAAAAAAADaAFm9QemZrM1M5UjhDYmlYZ1ZHZk5iM0E%3D&scroll=2m [status:404 request:0.006s]
2019-11-01 17:09:35,959 INFO pid=14670 tid=MainThread file=base.py:log_request_success:118 | DELETE http://mx12405vm:10212/_search/scroll [status:200 request:0.007s]
2019-11-01 17:09:35,960 ERROR pid=14670 tid=MainThread file=base_modinput.py:log_error:307 | Get error when collecting events.
Traceback (most recent call last):
File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/modinput_wrapper/base_modinput.py", line 127, in stream_events
self.collect_events(ew)
File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/elasticsearch_json.py", line 104, in collect_events
input_module.collect_events(self, ew)
File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/input_module_elasticsearch_json.py", line 83, in collect_events
for doc in res:
File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/helpers/actions.py", line 458, in scan
resp = client.scroll(**scroll_kwargs)
File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/client/utils.py", line 84, in wrapped
return func(*args, params=params, **kwargs)
File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/client/
init_.py", line 1315, in scroll
"GET", "/_search/scroll", params=params, body=body
File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/transport.py", line 353, in perform_request
timeout=timeout,
File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/connection/http_urllib3.py", line 251, in perform_request
self._raise_error(response.status, raw_data)
File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/connection/base.py", line 178, in _raise_error
status_code, error_message, additional_info
NotFoundError: NotFoundError(404, u'index_not_found_exception', u'no such index', bad-request, index_or_alias)

0 Karma

gaurav_maniar
Builder

as per the error message, the index you have specified in the configuration is not available in ES or Splunk.

Recheck the configuration values for any extra spaces.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...