All Apps and Add-ons
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Ask a Question

Distributed tracing from elastic search into Splunk using the “Elasticsearch Data Integrator”

robertlynch2020
Motivator
‎11-05-2019 06:47 AM

Hi @gaurav_maniar @larmesto (I am not sure your expect eye on python can help here, i hope it can)

I have downloaded the app and I have it working for traces into Splunk.
I can get standered JSON events into Splunk and all is good there, however when i have tired to change to Distributed tracing, i cant get the data to come into Splunk! I think it is due to the fact the timestamp is different in this new sourcetype. So i have tried to define a new sourcetype with new time, but getting errors.

With the following configuration, however when I tried to get the Distributed tracing to work it is not.

[elasticsearch_json://jaeger-span2]
date_field_name = startTime
elasticsearch_indice = jaeger-span-*
elasticsearch_instance_url = http://mx12405vm
greater_or_equal = 2019-01-01
index = mlc_test
interval = 10
lower_or_equal = now
port = 10212
use_ssl = False
verify_certs = False
user = 
secret = 
sourcetype = ta_elasticsearch
host = test123
disabled = 0

The errors i am gettign are the following.

Splunkd.log
11-05-2019 15:37:34.276 +0100 ERROR ExecProcessor - message from "/hp400srv2/apps/SPLUNK_8/splunk/bin/python2.7 /hp400srv2/apps/SPLUNK_8/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/elasticsearch_json.py"     status_code, error_message, additional_info
11-05-2019 15:37:34.276 +0100 ERROR ExecProcessor - message from "/hp400srv2/apps/SPLUNK_8/splunk/bin/python2.7 /hp400srv2/apps/SPLUNK_8/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/elasticsearch_json.py" RequestError: RequestError(400, u'search_phase_execution_exception', u'failed to create query: {\n  "bool" : {\n    "filter" : [\n      {\n        "range" : {\n          "startTime" : {\n            "from" : "2019-01-01",\n            "to" : "now",\n            "include_lower" : true,\n            "include_upper" : true,\n            "boost" : 1.0\n          }\n        }\n      }\n    ],\n    "disable_coord" : false,\n    "adjust_pure_negative" : true,\n    "boost" : 1.0\n  }\n}')

Elastic.log

Elastic.log
2019-11-05 15:38:41,776 ERROR pid=20450 tid=MainThread file=base_modinput.py:log_error:307 | Get error when collecting events.
Traceback (most recent call last):
  File "/hp400srv2/apps/SPLUNK_8/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/modinput_wrapper/base_modinput.py", line 127, in stream_events
    self.collect_events(ew)
  File "/hp400srv2/apps/SPLUNK_8/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/elasticsearch_json.py", line 104, in collect_events
    input_module.collect_events(self, ew)
  File "/hp400srv2/apps/SPLUNK_8/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/input_module_elasticsearch_json.py", line 83, in collect_events
    for doc in res:
  File "/hp400srv2/apps/SPLUNK_8/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/helpers/actions.py", line 435, in scan
    body=query, scroll=scroll, size=size, request_timeout=request_timeout, **kwargs
  File "/hp400srv2/apps/SPLUNK_8/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/client/utils.py", line 84, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/hp400srv2/apps/SPLUNK_8/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/client/__init__.py", line 819, in search
    "GET", _make_path(index, "_search"), params=params, body=body
  File "/hp400srv2/apps/SPLUNK_8/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/transport.py", line 353, in perform_request
    timeout=timeout,
  File "/hp400srv2/apps/SPLUNK_8/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/connection/http_urllib3.py", line 251, in perform_request
    self._raise_error(response.status, raw_data)
  File "/hp400srv2/apps/SPLUNK_8/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/connection/base.py", line 178, in _raise_error
    status_code, error_message, additional_info
RequestError: RequestError(400, u'search_phase_execution_exception', u'failed to create query: {\n  "bool" : {\n    "filter" : [\n      {\n        "range" : {\n          "startTime" : {\n            "from" : "2019-01-01",\n            "to" : "now",\n            "include_lower" : true,\n            "include_upper" : true,\n            "boost" : 1.0\n          }\n        }\n      }\n    ],\n    "disable_coord" : false,\n    "adjust_pure_negative" : true,\n    "boost" : 1.0\n  }\n}')

Example of the JSON i am trying to get out our elastic search. Starttime is the timestamp.
I have chreated a new one myself from some sample date and that work manually... Any help would be great cheers :

{
"_index": "jaeger-span-2019-11-01",
"_type": "span",
"_id": "AW4mrE4iQJeZAcmXbCYF",
"_version": 1,
"_score": 1,
"_source": {
"traceID": "d1daf2fd2f90b222",
"spanID": "2f6463e05c0b932d",
"flags": 1,
"operationName": "emit-tuple",
"references": [
{
"refType": "CHILD_OF",
"traceID": "d1daf2fd2f90b222",
"spanID": "b7e7335a859c15b4"
}
],
"startTime": 1572606855896000,
"startTimeMillis": 1572606855896,
"duration": 1514,
"tags": [ ],
"logs": [ ],
"process": {
"serviceName": "positions-storm-supervisor-v1",
"tags": [
{
"key": "hostname",
"type": "string",
"value": "mx12405vm"
}
,
{
"key": "jaeger.version",
"type": "string",
"value": "Java-0.32.0"
}
,
{
"key": "ip",
"type": "string",
"value": "10.26.10.130"
}
]
}
}
}
0 Karma
Reply

gaurav_maniar
Builder
‎11-07-2019 05:50 AM

Nor sure if the following changes will work or not, just give it try.

Open file - /hp400srv2/apps/SPLUNK_8/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/transport.py
- goto line number 353 and change it from timeout=timeout to timeout=60s
- add print str(body) before line number 341 try

Let me know the new errors

0 Karma
Reply

robertlynch2020
Motivator
‎11-07-2019 06:18 AM

I am not sure i have put this in correctly @gaurav_maniar 

      for attempt in range(self.max_retries + 1):
            connection = self.get_connection()


            print str(body)             
            try:
                # add a delay before attempting the next retry
                # 0, 1, 3, 7, etc...
                delay = 2 ** attempt - 1

11-07-2019 15:19:31.851 +0100 ERROR ExecProcessor - message from "/hp400srv2/apps/SPLUNK_8/splunk/bin/python2.7 /hp400srv2/apps/SPLUNK_8/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/elasticsearch_json.py" File "/hp400srv2/apps/SPLUNK_8/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/transport.py", line 342
11-07-2019 15:19:31.851 +0100 ERROR ExecProcessor - message from "/hp400srv2/apps/SPLUNK_8/splunk/bin/python2.7 /hp400srv2/apps/SPLUNK_8/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/elasticsearch_json.py" print str(body)
11-07-2019 15:19:31.851 +0100 ERROR ExecProcessor - message from "/hp400srv2/apps/SPLUNK_8/splunk/bin/python2.7 /hp400srv2/apps/SPLUNK_8/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/elasticsearch_json.py" ^
11-07-2019 15:19:31.851 +0100 ERROR ExecProcessor - message from "/hp400srv2/apps/SPLUNK_8/splunk/bin/python2.7 /hp400srv2/apps/SPLUNK_8/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/elasticsearch_json.py" IndentationError: unexpected indent

0 Karma
Reply

gaurav_maniar
Builder
‎11-07-2019 06:40 AM

just remove that print line and try with timeout changes

0 Karma
Reply

gaurav_maniar
Builder
‎11-08-2019 01:58 AM

it looks like the issue with epoc time format of Splunk & standard format in Elastic.
I don't know much about Elastic, but will let you know if I find something on this.

0 Karma
Reply

robertlynch2020
Motivator
‎11-08-2019 02:10 AM

Hi

I totally agree, so the question is how can i make Splunk look epoc in elastic

Working (For other event in Elasticsearch )
"timestamp": "2019-08-08T16:25:58.751Z"

not working due
startTime=1572876800520000,

0 Karma
Reply
Get Updates on the Splunk Community!

Announcing Scheduled Export GA for Dashboard Studio

We're excited to announce the general availability of Scheduled Export for Dashboard Studio. Starting in ...

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics GA in US-AWS!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...