Visibility and Monitoring for HEC base Data Ingestion Interruptions

[Nraj87]

please advise whether there is a solution or monitoring use case to identify interruptions in HEC base data ingestion.

Specifically:

• When the data ingestion service (HEC Token) becomes unavailable OR Down

• When the service (HEC Token) is operational, but no data/logs are being received during normal business hours.

[tscroggins]

Hi @Nraj87,

You can probe the services/collector/health endpoint on the HEC port for current service, ack, and token status. See https://help.splunk.com/en/splunk-enterprise/leverage-rest-apis/rest-api-reference/10.0/input-endpoi... or your version's documentation for more information.

HEC metrics are available in index=_introspection with sourcetype=http_event_collector_metrics, e.g.:

index=_introspection sourcetype=http_event_collector_metrics component=HttpEventCollector data.series=http_event_collector

index=_introspection sourcetype=http_event_collector_metrics component=HttpEventCollector data.series=http_event_collector_token

| tstats latest(data.num_of_events) as num_of_events latest(data.num_of_requests) as num_of_requests where index=_introspection sourcetype=http_event_collector_metrics component=HttpEventCollector data.series=http_event_collector

| tstats sum(data.num_of_events) as num_of_events sum(data.num_of_requests) as num_of_requests where index=_introspection sourcetype=http_event_collector_metrics component=HttpEventCollector data.series=http_event_collector_token data.token_name=foo by _time

Introspection metrics are relative to the report window, i.e., data.num_of_events is the number of events received over the last 60 seconds using the default limits.conf [http_input] stanza settings. Token-level introspection events are only generated when activity occurs over the report window. See https://help.splunk.com/en/splunk-enterprise/get-started/get-data-in/10.0/get-data-with-http-event-c... for more information.