Visibility and Monitoring for HEC base Data Ingestion Interruptions

[Nraj87]

please advise whether there is a solution or monitoring use case to identify interruptions in HEC base data ingestion.

Specifically:

• When the data ingestion service (HEC Token) becomes unavailable OR Down

• When the service (HEC Token) is operational, but no data/logs are being received during normal business hours.

[Nraj87]

Above Suggestion didn't work, Monitoring should generate an alert whenever the Splunk HEC data connection is disrupted or stops working to prevent data loss.

[tscroggins]

Hi @Nraj87,

I'm not AI.

Did you read the referenced documentation and experiment with health checks and thresholds?

In general:

1. Model your system's behavior in a controlled environment. What are its principal components (metrics)? What are the shapes/distributions of the metrics?

2. Measure your system's activity. What are the current values of the metrics?

3. Monitor (compare) your measurements to your model. Is the current metric value outside the range of acceptable values based on your model?

[tscroggins]

Hi @Nraj87,

You can probe the services/collector/health endpoint on the HEC port for current service, ack, and token status. See https://help.splunk.com/en/splunk-enterprise/leverage-rest-apis/rest-api-reference/10.0/input-endpoi... or your version's documentation for more information.

HEC metrics are available in index=_introspection with sourcetype=http_event_collector_metrics, e.g.:

index=_introspection sourcetype=http_event_collector_metrics component=HttpEventCollector data.series=http_event_collector

index=_introspection sourcetype=http_event_collector_metrics component=HttpEventCollector data.series=http_event_collector_token

| tstats latest(data.num_of_events) as num_of_events latest(data.num_of_requests) as num_of_requests where index=_introspection sourcetype=http_event_collector_metrics component=HttpEventCollector data.series=http_event_collector

| tstats sum(data.num_of_events) as num_of_events sum(data.num_of_requests) as num_of_requests where index=_introspection sourcetype=http_event_collector_metrics component=HttpEventCollector data.series=http_event_collector_token data.token_name=foo by _time

Introspection metrics are relative to the report window, i.e., data.num_of_events is the number of events received over the last 60 seconds using the default limits.conf [http_input] stanza settings. Token-level introspection events are only generated when activity occurs over the report window. See https://help.splunk.com/en/splunk-enterprise/get-started/get-data-in/10.0/get-data-with-http-event-c... for more information.