Dashboards & Visualizations

HTTP Event Collectors Invalid Token

eandresen
Path Finder

I am having issue with multiple sets of HTTP Event Collectors we have running, each of which are throwing a "{"text":"Invalid token","code":4}" message, as shown below, when I ran a simple curl command against them.

[root@ip-10-0-17-167 ~]# curl -k  https://<<EC_URL>>:8088/services/collector/event -H "Authorization: Splunk 297B4C96-5B44-44D2-A9C1-873862AAD558" -d '{"event": "hello world"}'
{"text":"Invalid token","code":4}

This is happening with several tokens, all of which were previously working without issues. The only thing that has changed that I am aware of since I last tested the functionality (at build out) was a minor upgrade from v6.3.3 to v6.3.9. With that said, I have tested both existing (pre-upgrade) and new (post-upgrade) tokens, both with same result.

We are using a Deployment server to generate the tokens from within the UI and deploy them out to the HTTP Event Collectors. On the Deployment server, all of the tokens are listed under the splunk_httpinput app, including the one I am using in the curl command provided above.

[root@ip-10-0-16-52 splunk_httpinput]# cat /opt/splunk/etc/deployment-apps/splunk_httpinput/local/inputs.conf
[http]
disabled = 0
port = 8088
enableSSL = 1
dedicatedIoThreads = 2
maxThreads = 0
maxSockets = 0

...

[http://adslot-lambda]
disabled = 0
index = app
sourcetype = adslot-lambda
token = 297B4C96-5B44-44D2-A9C1-873862AAD558

I also confrimed that the tokens, including the one I am using in the curl command provided above, are deployed to the HTTP Event Collector I am pointed to. It is listed under the splunk_httpinput app just like it is listed on the Deployment server and Splunk has picked up the inputs setting following the reload.

[root@ip-10-0-18-38 apps]# cat /opt/splunk/etc/apps/splunk_httpinput/local/inputs.conf
[http]
disabled = 0
port = 8088
enableSSL = 1
dedicatedIoThreads = 2
maxThreads = 0
maxSockets = 0

...

[http://adslot-lambda]
disabled = 0
index = app
sourcetype = adslot-lambda
token = 297B4C96-5B44-44D2-A9C1-873862AAD558

[root@ip-10-0-18-38 apps]# /opt/splunk/bin/splunk cmd btool inputs --debug list

...

/opt/splunk/etc/apps/splunk_httpinput/local/inputs.conf                [http://adslot-lambda]
/opt/splunk/etc/system/default/inputs.conf                             _rcvbuf = 1572864
/opt/splunk/etc/apps/splunk_httpinput/local/inputs.conf                disabled = 0
/opt/splunk/etc/system/local/inputs.conf                               host = ip-10-0-18-38
/opt/splunk/etc/apps/splunk_httpinput/local/inputs.conf                index = app
/opt/splunk/etc/apps/splunk_httpinput/local/inputs.conf                sourcetype = adslot-lambda
/opt/splunk/etc/apps/splunk_httpinput/local/inputs.conf                token = 297B4C96-5B44-44D2-A9C1-873862AAD558

Please let me know if additional informaiton is needed and thanks in advanced for any assistance you can provide me.

0 Karma
1 Solution

starcher
Influencer

Make sure useDeploymentServer is not in the config in your heavy forwarders.

View solution in original post

KrishatSplunk
Loves-to-Learn

If you are using deploment server to create the token and push it to your heavy forwarders where it should be actually authenticate then you have to:
1. To make sure you change useDeploymentServer flag to true as below.

 

useDeploymentServer = 1

 

When this option is set to 1 and you make UI-based HEC changes on the deployment server, those changes are placed directly in the $SPLUNK_HOME/etc/deployment-apps/splunk_httpinput/ folder, rather than in $SPLUNK_HOME/etc/apps/folder. 

Because if  your inputs changes is there in the $SPLUNK_HOME/etc/apps/<anyapp>/inputs.conf  on deployment server and also in your Heavy forwarder . Then the rest/curl call to token will end up in Invalid token response code 4.

0 Karma

BigDaddyPayne
New Member

Any insights? I still have this issue. Thanks

0 Karma

starcher
Influencer

Make sure useDeploymentServer is not in the config in your heavy forwarders.

eandresen
Path Finder

To make sure I did not screw anything up that I am not aware with our existing/broken Event Collectors, I started with a new setup and I am still having the same problems. I used the process outlined on Splunk Doc's website: http://docs.splunk.com/Documentation/Splunk/latest/Data/UsetheHTTPEventCollector

On the Deployment server I copied the "splunk_httpinput" app from the apps folder into the deployment-apps folder. From the Deployment server's UI, I enabled the EC service and checked the "Use Deployment Server" option. I also created a test token and it along with the settings are showing up in the "splunk_httpinput" app.

[root@ip-10-0-18-55 ~]# cat /opt/splunk/etc/deployment-apps/splunk_httpinput/default/inputs.conf
[http]
useDeploymentServer = 1
disabled = 0
port = 8088
enableSSL = 1
dedicatedIoThreads = 2
maxThreads = 0
maxSockets = 0

[root@ip-10-0-18-55 ~]# cat /opt/splunk/etc/deployment-apps/splunk_httpinput/local/inputs.conf
[http]
_rcvbuf = 1572864
allowSslCompression = true
allowSslRenegotiation = true
host = ip-10-0-18-55
index = app
sslVersions = *,-ssl2
enableSSL = 1

[http://Splunk EWE Admins]
disabled = 0
host = ip-10-0-18-55
index = app
sourcetype = splunkeweadmins
token = 04BD0B3D-A37C-4403-80E0-CDF37F5E9892

I setup the serverclass.conf file on the Deployment server to deploy the "splunk_httpinput" app, along with our outputs.conf app, to the one Event Collector. Both apps, along with the test token are now showing on the Event Collector.

[root@ip-10-0-18-122 ~]# cat /opt/splunk/etc/apps/splunk_httpinput/default/inputs.conf
[http]
useDeploymentServer = 1
disabled = 0
port = 8088
enableSSL = 1
dedicatedIoThreads = 2
maxThreads = 0
maxSockets = 0

[root@ip-10-0-18-122 ~]# cat /opt/splunk/etc/apps/splunk_httpinput/local/inputs.conf
[http]
_rcvbuf = 1572864
allowSslCompression = true
allowSslRenegotiation = true
host = ip-10-0-18-55
index = app
sslVersions = *,-ssl2
enableSSL = 1

[http://Splunk EWE Admins]
disabled = 0
host = ip-10-0-18-55
index = app
sourcetype = splunkeweadmins
token = 04BD0B3D-A37C-4403-80E0-CDF37F5E9892

However, when I send a test event using curl and my test token, I still get the ""Invalid token" error message and nothing indexed into the Splunk environment.

[root@ip-10-0-18-55 ~]# curl -k  https://<<ec_endpoint>>:8088/services/collector/event -H "Authorization: Splunk 04BD0B3D-A37C-4403-80E0-CDF37F5E9892" -d '{"event": "hello world"}'
{"text":"Invalid token","code":4}

Any other thoughts? Is there something wrong with my setup or configuration?

0 Karma
Get Updates on the Splunk Community!

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...