HEC - verify what sources are using a HEC token

[stevensk]

We want to be able to monitor what sources/devices are using what HEC tokens.

I know we can use _introspection to retrieve metrics on a HEC token to see if it is being used, but need to know "what" is sending to/using a HEC token.  What sources (IP or host) are sending to a HEC token.

We are using Splunk Cloud. 

[livehybrid]

Hi @Cievo - please can you reference the documentation relating to "enableSplunkdSidechannel"? This isnt something that I am familiar with.

@kiran_panchavat The information on "you must download the ACS Open API 3.0 specification" is incorrect in relation to this? This does not relate to the question and is not a requirement for using ACS? 

@stevensk Whilst I'm not able to give the answer you might want, I do not believe its possible to see a per-source breakdown for data received from HEC. I have tried this in the past and it led nowhere, however hopefully someone here might be able to shine more light on it. You can see metrics in "index=_introspection sourcetype=http_event_collector_metrics component=HttpEventCollector" which will give info at per-HEC-token level - however there arent logs which combine a metric for HEC Token and source.

The only other way to approach this would be to improve segregation with your HEC tokens so that you can maybe search specific indexes/sourcetype/sources for data you know comes from a specific HEC token, OR add a custom field into the HEC payload which you can then use to determine the metrics you need (this is what I ended up doing!)

Good luck 🙂

Please let me know how you get on and consider accepting this answer or adding karma this answer if it has helped.
Regards

Will

[stevensk]

Thanks @livehybrid, so there does not seem to be a solution.

The issue we are trying to address is if someone gets their hands on a HEC token, they could just send data to our Splunk Cloud instance via that HEC token. We have set specific indexes for specific tokens, so that should limit this, but just trying to find a way to identify what is sending to a specific HEC token so we can monitor this.

Do you have anymore info on how you added a custom field into the HEC payload?

[isoutamo]

Hi
I just found this https://community.splunk.com/t5/Getting-Data-In/How-to-configure-HTTP-Event-collector-to-log-client-...
I haven’t checked if this is valid also on SCP or not?
r. Ismo

[livehybrid]

Hi @stevensk 

Sorry I missed this reply - leave it with me and let me see if I can dig out what we did (it was a few years ago!)

[kiran_panchavat]

@stevensk 

Before using the ACS API, you must download the ACS Open API 3.0 specification, which includes the parameters, codes, and other data you need to work with the ACS API. You must also create an authentication token in Splunk Cloud Platform for use with ACS endpoint requests. For details on how to set up the ACS API, see 

https://docs.splunk.com/Documentation/SplunkCloud/9.3.2408/Config/ACSusage#Set_up_the_ACS_API 

https://docs.splunk.com/Documentation/SplunkCloud/9.3.2408/Config/ManageHECtokens?_gl=1*68bsg2*_gcl_... 

[kiran_panchavat]

@stevensk 

You could get configured HEC tokens/inputs from HEC node e.g.

| rest splunk_server=<your hec node> /services/data/inputs/http

Of course you should have added that node to peer your SH or just run above towards your HEC node(s) with curl.

That query shows allowed indexes and forced indexes for those tokens.

Another way to check which tokens are used is

You can check which HEC token is in use in _introspection Index with below query.

index=_introspection host=YOUR_HEC_HOST  sourcetype=http_event_collector_metrics data.token_name=*
| rename data.* as *
| table host, component, token_name, num_*

If there will be 0 num_of_requests or num_of_events for longer time span then I guess you can disable those token for few days and then remove it.

[Cievo]

Hello @stevensk 

To be able to see the source IP of HEC forwarders you need to enable logging on HEC's input.conf file:

[http]
enableSplunkdSidechannel = true

And then run search to see logs containing specific token:

index=_internal sourcetype=splunkd "token="

To filter by source IP you can run for example this search:

index=_internal sourcetype=splunkd "token="
| rex "sourceIp=(?<source_ip>\d+\.\d+\.\d+\.\d+)"
| stats count by source_ip

I hope this will help you.

Have a nice day,

[stevensk]

Hi

Thanks for the info.

This is Splunk Cloud so we cannot edit any conf files, nor is there an option in the Web UI when creating HEC tokens to enable this.

The following search seems to give all Errors for devices trying to connect with a HEC token, but I do not seem to see successful sources, only failed.

index=_internal sourcetype=splunkd component=HttpInputDataHandler

Also the source_IP value, since it is Splunk Cloud, are the Splunk Cloud Loadbalancer IPs. We were told this in a case with Splunk.

[kiran_panchavat]

@stevensk 

To monitor which sources or devices are using specific HEC tokens in Splunk Cloud, you can leverage the Admin Config Service (ACS) API. Here's a high-level overview of how you can achieve this:

https://docs.splunk.com/Documentation/SplunkCloud/9.3.2408/Config/ManageHECtokens 

[stevensk]

Sorry but the ACS API will not help in this case.