All Apps and Add-ons
cancel
Turn on suggestions
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Ask a Question
Solved! Jump to solution

How to configure the Elasticsearch Data INtegrator

robertlynch2020
Influencer
‎10-31-2019 04:45 AM

Hi

In our company some logs are put to Elasticsearch, i have downloaded the Elasticsearch Data INtegrator to see if i can pull out the data.

When i log into Grafana i can see the configuration on how it access Elasticsearch

alt text

However when i try to reproduce it, i get errors.
alt text

ta_elasticsearch_data_integrator_modular_input_elasticsearch_json.log
2019-10-31 12:17:20,493 INFO pid=29104 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1
2019-10-31 12:17:22,408 INFO pid=29104 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1
2019-10-31 12:17:26,011 INFO pid=29104 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1
2019-10-31 12:17:29,643 ERROR pid=29104 tid=MainThread file=base_modinput.py:log_error:307 | Get error when collecting events.
Traceback (most recent call last):
File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/modinput_wrapper/base_modinput.py", line 127, in stream_events
self.collect_events(ew)
File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/elasticsearch_json.py", line 104, in collect_events
input_module.collect_events(self, ew)
File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/input_module_elasticsearch_json.py", line 49, in collect_events
opt_ca_certs_path = opt_ca_certs_path.strip()
AttributeError: 'NoneType' object has no attribute 'strip'

OR
splunks.log

10-31-2019 12:15:00.828 +0100 ERROR ExecProcessor - message from "python /hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/elasticsearch_json.py" Traceback (most recent call last):

I am new to Elasticsearch, so perhaps i am missing something easy? Any help would be great thanks

Rob

0 Karma
Reply
1 Solution
Solved! Jump to solution
Solution

gaurav_maniar
Builder
‎11-01-2019 05:07 AM

Hi @robertlynch2020 ,

The issue is related to CA Certificate path variable in the application script.
I don't have exact solution to your problem, but we can do trial & error check.

First, this is python variable value related error.
Now open python file - /hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/input_module_elasticsearch_json.py
and comment the line number 49 (put '#' at start of the line to convert it to comment)

Once above changes are done check the logs and let me know what the new errors you are getting.
(no need to restart splunk for python scripts related changes)

View solution in original post

0 Karma
Reply
Solved! Jump to solution
Solution

gaurav_maniar
Builder
‎11-01-2019 05:07 AM

Hi @robertlynch2020 ,

The issue is related to CA Certificate path variable in the application script.
I don't have exact solution to your problem, but we can do trial & error check.

First, this is python variable value related error.
Now open python file - /hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/input_module_elasticsearch_json.py
and comment the line number 49 (put '#' at start of the line to convert it to comment)

Once above changes are done check the logs and let me know what the new errors you are getting.
(no need to restart splunk for python scripts related changes)

0 Karma
Reply
Solved! Jump to solution

robertlynch2020
Influencer
‎11-01-2019 05:19 AM

cheers for help @gaurav_maniar

FYI This is my current configuration. I have made the changes you said and i disablede and enabled the connection. I am getting the following now.

 #opt_ca_certs_path = opt_ca_certs_path.strip()

[elasticsearch_json://esearch]
date_field_name = body.ts
elasticsearch_indice = metric-2019.11.01
elasticsearch_instance_url = http://mx12405vm
greater_or_equal = 2019-01-01
index = mlc_test
interval = 60
lower_or_equal = now
port = 10212
use_ssl = False
verify_certs = False
user =
secret =
sourcetype = elasticsearch


2019-11-01 13:17:01,163 INFO pid=13710 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1
2019-11-01 13:17:03,063 INFO pid=13710 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1
2019-11-01 13:17:06,505 INFO pid=13710 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1
2019-11-01 13:17:09,953 INFO pid=13710 tid=MainThread file=setup_util.py:log_info:114 | Log level is not set, use default INFO
2019-11-01 13:17:09,954 WARNING pid=13710 tid=MainThread file=base.py:log_request_fail:146 | GET http://mx12405vm:10212/metric-2019.11.01/_search?scroll=2m&size=1000 [status:N/A request:0.000s]
Traceback (most recent call last):
  File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/connection/http_urllib3.py", line 232, in perform_request
    method, url, body, retries=Retry(False), headers=request_headers, **kw
  File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/urllib3/util/retry.py", line 343, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/urllib3/connectionpool.py", line 354, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/hp737srv2/apps/splunk/lib/python2.7/httplib.py", line 1042, in request
    self._send_request(method, url, body, headers)
  File "/hp737srv2/apps/splunk/lib/python2.7/httplib.py", line 1082, in _send_request
    self.endheaders(body)
  File "/hp737srv2/apps/splunk/lib/python2.7/httplib.py", line 1038, in endheaders
    self._send_output(message_body)
  File "/hp737srv2/apps/splunk/lib/python2.7/httplib.py", line 882, in _send_output
    self.send(msg)
  File "/hp737srv2/apps/splunk/lib/python2.7/httplib.py", line 844, in send
    self.connect()
  File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/urllib3/connection.py", line 196, in connect
    conn = self._new_conn()
  File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/urllib3/connection.py", line 180, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fce6168fd10>: Failed to establish a new connection: getaddrinfo() argument 2 must be integer or string
2019-11-01 13:17:10,955 WARNING pid=13710 tid=MainThread file=base.py:log_request_fail:146 | GET http://mx12405vm:10212/metric-2019.11.01/_search?scroll=2m&size=1000 [status:N/A request:0.000s]
Traceback (most recent call last):
  File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/connection/http_urllib3.py", line 232, in perform_request
    method, url, body, retries=Retry(False), headers=request_headers, **kw
  File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/urllib3/util/retry.py", line 343, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/urllib3/connectionpool.py", line 354, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/hp737srv2/apps/splunk/lib/python2.7/httplib.py", line 1042, in request
    self._send_request(method, url, body, headers)
  File "/hp737srv2/apps/splunk/lib/python2.7/httplib.py", line 1082, in _send_request
    self.endheaders(body)
  File "/hp737srv2/apps/splunk/lib/python2.7/httplib.py", line 1038, in endheaders
    self._send_output(message_body)
  File "/hp737srv2/apps/splunk/lib/python2.7/httplib.py", line 882, in _send_output
    self.send(msg)
  File "/hp737srv2/apps/splunk/lib/python2.7/httplib.py", line 844, in send
    self.connect()
  File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/urllib3/connection.py", line 196, in connect
    conn = self._new_conn()
  File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/urllib3/connection.py", line 180, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fce6168ff10>: Failed to establish a new connection: getaddrinfo() argument 2 must be integer or string
^C
0 Karma
Reply
Solved! Jump to solution

gaurav_maniar
Builder
‎11-01-2019 08:38 AM

Again python related error.

now open python file - /hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/urllib3/connection.py

Goto line number 171,

(self._dns_host, self.port), self.timeout, **extra_kw)

change it to following, changing port from unicode to string,

(self._dns_host, self.port.encode('ascii')), self.timeout, **extra_kw)

Previously it worked on windows, give it a try on linux.

0 Karma
Reply
Solved! Jump to solution

robertlynch2020
Influencer
‎11-01-2019 08:47 AM

Hi @gaurav_maniar

It now works now that i update the date_field_name = timestamp.
Thanks so so much 🙂

[elasticsearch_json://esearch]
date_field_name = timestamp
elasticsearch_indice = metric-2019.11.01
elasticsearch_instance_url = http://mx12405vm
greater_or_equal = 2019-01-01
index = mlc_test
interval = 60
lower_or_equal = now
port = 10212
use_ssl = False
verify_certs = False
user =
secret =
sourcetype = elasticsearch

0 Karma
Reply
Solved! Jump to solution

Chika
Loves-to-Learn Lots
‎12-06-2020 08:50 PM

The Elasticsearch Data Integrator appears to be not responsive when a non-admin user tries to access the inputs and configuration menu. Any help why this is happening?

0 Karma
Reply
Solved! Jump to solution

robertlynch2020
Influencer
‎11-01-2019 09:11 AM

Just to add i am still gettting errors but the basic connection is thre.

OVI4Q2JpWGdWR2ZOYjNBAAAAAAAANn8Wb1B6ZmszUzlSOENiaVhnVkdmTmIzQQAAAAAAADaAFm9QemZrM1M5UjhDYmlYZ1ZHZk5iM0E%3D&scroll=2m [status:404 request:0.006s]
2019-11-01 17:09:35,959 INFO pid=14670 tid=MainThread file=base.py:log_request_success:118 | DELETE http://mx12405vm:10212/_search/scroll [status:200 request:0.007s]
2019-11-01 17:09:35,960 ERROR pid=14670 tid=MainThread file=base_modinput.py:log_error:307 | Get error when collecting events.
Traceback (most recent call last):
File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/modinput_wrapper/base_modinput.py", line 127, in stream_events
self.collect_events(ew)
File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/elasticsearch_json.py", line 104, in collect_events
input_module.collect_events(self, ew)
File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/input_module_elasticsearch_json.py", line 83, in collect_events
for doc in res:
File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/helpers/actions.py", line 458, in scan
resp = client.scroll(**scroll_kwargs)
File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/client/utils.py", line 84, in wrapped
return func(*args, params=params, **kwargs)
File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/client/init_.py", line 1315, in scroll
"GET", "/_search/scroll", params=params, body=body
File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/transport.py", line 353, in perform_request
timeout=timeout,
File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/connection/http_urllib3.py", line 251, in perform_request
self._raise_error(response.status, raw_data)
File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/connection/base.py", line 178, in _raise_error
status_code, error_message, additional_info
NotFoundError: NotFoundError(404, u'index_not_found_exception', u'no such index', bad-request, index_or_alias)

0 Karma
Reply
Solved! Jump to solution

gaurav_maniar
Builder
‎11-01-2019 11:21 AM

as per the error message, the index you have specified in the configuration is not available in ES or Splunk.

Recheck the configuration values for any extra spaces.

0 Karma
Reply
Get Updates on the Splunk Community!

Mastering Data Pipelines: Unlocking Value with Splunk

 In today's AI-driven world, organizations must balance the challenges of managing the explosion of data with ...

The Latest Cisco Integrations With Splunk Platform!

Join us for an exciting tech talk where we’ll explore the latest integrations in Cisco &#43; Splunk! We’ve ...

AI Adoption Hub Launch | Curated Resources to Get Started with AI in Splunk

Hey Splunk Practitioners and AI Enthusiasts! It’s no secret (or surprise) that AI is at the forefront of ...
Top