Distributed tracing from elastic search into Splunk using the “Elasticsearch Data Integrator”

[robertlynch2020]

Hi @gaurav_maniar @larmesto (I am not sure your expect eye on python can help here, i hope it can)

I have downloaded the app and I have it working for traces into Splunk.
I can get standered JSON events into Splunk and all is good there, however when i have tired to change to Distributed tracing, i cant get the data to come into Splunk! I think it is due to the fact the timestamp is different in this new sourcetype. So i have tried to define a new sourcetype with new time, but getting errors.

With the following configuration, however when I tried to get the Distributed tracing to work it is not.

[elasticsearch_json://jaeger-span2]
date_field_name = startTime
elasticsearch_indice = jaeger-span-*
elasticsearch_instance_url = http://mx12405vm
greater_or_equal = 2019-01-01
index = mlc_test
interval = 10
lower_or_equal = now
port = 10212
use_ssl = False
verify_certs = False
user = 
secret = 
sourcetype = ta_elasticsearch
host = test123
disabled = 0

The errors i am gettign are the following.

Splunkd.log
11-05-2019 15:37:34.276 +0100 ERROR ExecProcessor - message from "/hp400srv2/apps/SPLUNK_8/splunk/bin/python2.7 /hp400srv2/apps/SPLUNK_8/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/elasticsearch_json.py"     status_code, error_message, additional_info
11-05-2019 15:37:34.276 +0100 ERROR ExecProcessor - message from "/hp400srv2/apps/SPLUNK_8/splunk/bin/python2.7 /hp400srv2/apps/SPLUNK_8/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/elasticsearch_json.py" RequestError: RequestError(400, u'search_phase_execution_exception', u'failed to create query: {\n  "bool" : {\n    "filter" : [\n      {\n        "range" : {\n          "startTime" : {\n            "from" : "2019-01-01",\n            "to" : "now",\n            "include_lower" : true,\n            "include_upper" : true,\n            "boost" : 1.0\n          }\n        }\n      }\n    ],\n    "disable_coord" : false,\n    "adjust_pure_negative" : true,\n    "boost" : 1.0\n  }\n}')

Elastic.log

Elastic.log
2019-11-05 15:38:41,776 ERROR pid=20450 tid=MainThread file=base_modinput.py:log_error:307 | Get error when collecting events.
Traceback (most recent call last):
  File "/hp400srv2/apps/SPLUNK_8/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/modinput_wrapper/base_modinput.py", line 127, in stream_events
    self.collect_events(ew)
  File "/hp400srv2/apps/SPLUNK_8/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/elasticsearch_json.py", line 104, in collect_events
    input_module.collect_events(self, ew)
  File "/hp400srv2/apps/SPLUNK_8/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/input_module_elasticsearch_json.py", line 83, in collect_events
    for doc in res:
  File "/hp400srv2/apps/SPLUNK_8/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/helpers/actions.py", line 435, in scan
    body=query, scroll=scroll, size=size, request_timeout=request_timeout, **kwargs
  File "/hp400srv2/apps/SPLUNK_8/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/client/utils.py", line 84, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/hp400srv2/apps/SPLUNK_8/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/client/__init__.py", line 819, in search
    "GET", _make_path(index, "_search"), params=params, body=body
  File "/hp400srv2/apps/SPLUNK_8/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/transport.py", line 353, in perform_request
    timeout=timeout,
  File "/hp400srv2/apps/SPLUNK_8/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/connection/http_urllib3.py", line 251, in perform_request
    self._raise_error(response.status, raw_data)
  File "/hp400srv2/apps/SPLUNK_8/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/connection/base.py", line 178, in _raise_error
    status_code, error_message, additional_info
RequestError: RequestError(400, u'search_phase_execution_exception', u'failed to create query: {\n  "bool" : {\n    "filter" : [\n      {\n        "range" : {\n          "startTime" : {\n            "from" : "2019-01-01",\n            "to" : "now",\n            "include_lower" : true,\n            "include_upper" : true,\n            "boost" : 1.0\n          }\n        }\n      }\n    ],\n    "disable_coord" : false,\n    "adjust_pure_negative" : true,\n    "boost" : 1.0\n  }\n}')

Example of the JSON i am trying to get out our elastic search. Starttime is the timestamp.
I have chreated a new one myself from some sample date and that work manually... Any help would be great cheers :

{
"_index": "jaeger-span-2019-11-01",
"_type": "span",
"_id": "AW4mrE4iQJeZAcmXbCYF",
"_version": 1,
"_score": 1,
"_source": {
"traceID": "d1daf2fd2f90b222",
"spanID": "2f6463e05c0b932d",
"flags": 1,
"operationName": "emit-tuple",
"references": [
{
"refType": "CHILD_OF",
"traceID": "d1daf2fd2f90b222",
"spanID": "b7e7335a859c15b4"
}
],
"startTime": 1572606855896000,
"startTimeMillis": 1572606855896,
"duration": 1514,
"tags": [ ],
"logs": [ ],
"process": {
"serviceName": "positions-storm-supervisor-v1",
"tags": [
{
"key": "hostname",
"type": "string",
"value": "mx12405vm"
}
,
{
"key": "jaeger.version",
"type": "string",
"value": "Java-0.32.0"
}
,
{
"key": "ip",
"type": "string",
"value": "10.26.10.130"
}
]
}
}
}

[gaurav_maniar]

Nor sure if the following changes will work or not, just give it try.

Open file - /hp400srv2/apps/SPLUNK_8/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/transport.py
- goto line number 353 and change it from timeout=timeout to timeout=60s
- add print str(body) before line number 341 try

Let me know the new errors

[robertlynch2020]

I am not sure i have put this in correctly @gaurav_maniar

      for attempt in range(self.max_retries + 1):
            connection = self.get_connection()


            print str(body)             
            try:
                # add a delay before attempting the next retry
                # 0, 1, 3, 7, etc...
                delay = 2 ** attempt - 1

11-07-2019 15:19:31.851 +0100 ERROR ExecProcessor - message from "/hp400srv2/apps/SPLUNK_8/splunk/bin/python2.7 /hp400srv2/apps/SPLUNK_8/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/elasticsearch_json.py" File "/hp400srv2/apps/SPLUNK_8/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/transport.py", line 342
11-07-2019 15:19:31.851 +0100 ERROR ExecProcessor - message from "/hp400srv2/apps/SPLUNK_8/splunk/bin/python2.7 /hp400srv2/apps/SPLUNK_8/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/elasticsearch_json.py" print str(body)
11-07-2019 15:19:31.851 +0100 ERROR ExecProcessor - message from "/hp400srv2/apps/SPLUNK_8/splunk/bin/python2.7 /hp400srv2/apps/SPLUNK_8/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/elasticsearch_json.py" ^
11-07-2019 15:19:31.851 +0100 ERROR ExecProcessor - message from "/hp400srv2/apps/SPLUNK_8/splunk/bin/python2.7 /hp400srv2/apps/SPLUNK_8/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/elasticsearch_json.py" IndentationError: unexpected indent

[gaurav_maniar]

just remove that print line and try with timeout changes

[gaurav_maniar]

it looks like the issue with epoc time format of Splunk & standard format in Elastic.
I don't know much about Elastic, but will let you know if I find something on this.

[robertlynch2020]

Hi

I totally agree, so the question is how can i make Splunk look epoc in elastic

Working (For other event in Elasticsearch )
"timestamp": "2019-08-08T16:25:58.751Z"

not working due
startTime=1572876800520000,