All Apps and Add-ons

Setting host and sourcetype in elasticsearch-data-integrator


Hi @larmesto

I have the following configuration that is working and taking in data into SPLUNK from elasticsearch via TA-elasticsearch-data-integrator.
The issues is, i cant specify the host or the sourcetype..
By deafult Sourcetype= JSON and host is the name of the Splunk machine. I want to change this as i need to connect to many elasticsearch .

date_field_name = timestamp
elasticsearch_indice = metric-*
elasticsearch_instance_url = http://mx12405vm
greater_or_equal = 2019-01-01
index = mlc_test
interval = 60
lower_or_equal = now
port = 10212
use_ssl = False
verify_certs = False
user = 
secret = 
sourcetype = Metric_Elastic
disabled = 0

alt text

I am getting an error, however the data is going into the correct index. So not sure if related.

2019-11-04 14:41:59,052 INFO pid=11049 tid=MainThread | DELETE http://mx12405vm:10212/_search/scroll [status:200 request:0.014s]
2019-11-04 14:41:59,053 ERROR pid=11049 tid=MainThread | Get error when collecting events.
Traceback (most recent call last):
  File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/modinput_wrapper/", line 127, in stream_events
  File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/", line 104, in collect_events
    input_module.collect_events(self, ew)
  File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/", line 83, in collect_events
    for doc in res:
  File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/helpers/", line 458, in scan
    resp = client.scroll(**scroll_kwargs)
  File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/client/", line 84, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/client/", line 1315, in scroll
    "GET", "/_search/scroll", params=params, body=body
  File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/", line 353, in perform_request
  File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/connection/", line 251, in perform_request
    self._raise_error(response.status, raw_data)
  File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/connection/", line 178, in _raise_error
    status_code, error_message, additional_info
NotFoundError: NotFoundError(404, u'index_not_found_exception', u'no such index', bad-request, index_or_alias)

You have new mail in /var/spool/mail/autoengine
0 Karma

Path Finder

I think the inability to set your own sourcetype is a bug. But there is workaround. Use source for set sourcetype and host. Insert TRANSFORMS in your props.conf and add the transformation to transforms.conf


KV_MODE = json
TIME_PREFIX = "@timestamp":
TRANSFORMS-host_sourcetype_override = elk_host_override, elk_sourcetype_override


REGEX = "beat":\s*{[^}]*?"hostname":\s*"([^"]+)
DEST_KEY = MetaData:Host
FORMAT = host::$1

DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::Metric_Elastic
0 Karma
Get Updates on the Splunk Community!

Tips & Tricks When Using Ingest Actions

Tune in to learn about:Large scale architecture when using Ingest ActionsRegEx performance considerations ...

Announcing Our Splunk MVPs

We are excited to announce the first cohort of the Splunk MVP program. Splunk MVPs are passionate members of ...

Dashboard Studio Challenge - Learn New Tricks, Showcase Your Skills, and Win Prizes!

Reimagine what you can do with your dashboards. Dashboard Studio is Splunk’s newest dashboard builder to ...