All Apps and Add-ons

Setting host and sourcetype in elasticsearch-data-integrator


Hi @larmesto

I have the following configuration that is working and taking in data into SPLUNK from elasticsearch via TA-elasticsearch-data-integrator.
The issues is, i cant specify the host or the sourcetype..
By deafult Sourcetype= JSON and host is the name of the Splunk machine. I want to change this as i need to connect to many elasticsearch .

date_field_name = timestamp
elasticsearch_indice = metric-*
elasticsearch_instance_url = http://mx12405vm
greater_or_equal = 2019-01-01
index = mlc_test
interval = 60
lower_or_equal = now
port = 10212
use_ssl = False
verify_certs = False
user = 
secret = 
sourcetype = Metric_Elastic
disabled = 0

alt text

I am getting an error, however the data is going into the correct index. So not sure if related.

2019-11-04 14:41:59,052 INFO pid=11049 tid=MainThread | DELETE http://mx12405vm:10212/_search/scroll [status:200 request:0.014s]
2019-11-04 14:41:59,053 ERROR pid=11049 tid=MainThread | Get error when collecting events.
Traceback (most recent call last):
  File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/modinput_wrapper/", line 127, in stream_events
  File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/", line 104, in collect_events
    input_module.collect_events(self, ew)
  File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/", line 83, in collect_events
    for doc in res:
  File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/helpers/", line 458, in scan
    resp = client.scroll(**scroll_kwargs)
  File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/client/", line 84, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/client/", line 1315, in scroll
    "GET", "/_search/scroll", params=params, body=body
  File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/", line 353, in perform_request
  File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/connection/", line 251, in perform_request
    self._raise_error(response.status, raw_data)
  File "/hp737srv2/apps/splunk/etc/apps/TA-elasticsearch-data-integrator---modular-input/bin/ta_elasticsearch_data_integrator_modular_input/elasticsearch/connection/", line 178, in _raise_error
    status_code, error_message, additional_info
NotFoundError: NotFoundError(404, u'index_not_found_exception', u'no such index', bad-request, index_or_alias)

You have new mail in /var/spool/mail/autoengine
0 Karma

Path Finder

I think the inability to set your own sourcetype is a bug. But there is workaround. Use source for set sourcetype and host. Insert TRANSFORMS in your props.conf and add the transformation to transforms.conf


KV_MODE = json
TIME_PREFIX = "@timestamp":
TRANSFORMS-host_sourcetype_override = elk_host_override, elk_sourcetype_override


REGEX = "beat":\s*{[^}]*?"hostname":\s*"([^"]+)
DEST_KEY = MetaData:Host
FORMAT = host::$1

DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::Metric_Elastic
0 Karma
Get Updates on the Splunk Community!

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...