All Apps and Add-ons

Setting host and sourcetype in elasticsearch-data-integrator

robertlynch2020
Motivator

Hi @larmesto

I have the following configuration that is working and taking in data into SPLUNK from elasticsearch via TA-elasticsearch-data-integrator.
The issues is, i cant specify the host or the sourcetype..
By deafult Sourcetype= JSON and host is the name of the Splunk machine. I want to change this as i need to connect to many elasticsearch .

[elasticsearch_json://metric]
date_field_name = timestamp
elasticsearch_indice = metric-*
elasticsearch_instance_url = http://mx12405vm
greater_or_equal = 2019-01-01
index = mlc_test
interval = 60
lower_or_equal = now
port = 10212
use_ssl = False
verify_certs = False
user = 
secret = 
sourcetype = Metric_Elastic
disabled = 0

alt text

I am getting an error, however the data is going into the correct index. So not sure if related.

2019-11-04 14:41:59,052 INFO pid=11049 tid=MainThread file=base.py:log_request_success:118 | DELETE http://mx12405vm:10212/_search/scroll [status:200 request:0.014s]
2019-11-04 14:41:59,053 ERROR pid=11049 tid=MainThread file=base_modinput.py:log_error:307 | Get error when collecting events.
Traceback (most recent call last):
  File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/modinput_wrapper/base_modinput.py", line 127, in stream_events
    self.collect_events(ew)
  File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/elasticsearch_json.py", line 104, in collect_events
    input_module.collect_events(self, ew)
  File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/input_module_elasticsearch_json.py", line 83, in collect_events
    for doc in res:
  File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/helpers/actions.py", line 458, in scan
    resp = client.scroll(**scroll_kwargs)
  File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/client/utils.py", line 84, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/client/__init__.py", line 1315, in scroll
    "GET", "/_search/scroll", params=params, body=body
  File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/transport.py", line 353, in perform_request
    timeout=timeout,
  File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/connection/http_urllib3.py", line 251, in perform_request
    self._raise_error(response.status, raw_data)
  File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/connection/base.py", line 178, in _raise_error
    status_code, error_message, additional_info
NotFoundError: NotFoundError(404, u'index_not_found_exception', u'no such index', bad-request, index_or_alias)

^C
You have new mail in /var/spool/mail/autoengine
0 Karma

BigCosta
Path Finder

Hi!
I think the inability to set your own sourcetype is a bug. But there is workaround. Use source for set sourcetype and host. Insert TRANSFORMS in your props.conf and add the transformation to transforms.conf

props.conf

[source::metric-*]
SHOULD_LINEMERGE = 0
KV_MODE = json
TIME_PREFIX = "@timestamp":
MAX_TIMESTAMP_LOOKAHEAD = 40
TRANSFORMS-host_sourcetype_override = elk_host_override, elk_sourcetype_override

transforms.conf

[elk_host_override]
REGEX = "beat":\s*{[^}]*?"hostname":\s*"([^"]+)
DEST_KEY = MetaData:Host
FORMAT = host::$1

[elk_sourcetype_override]
REGEX = .
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::Metric_Elastic
0 Karma
Get Updates on the Splunk Community!

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...