Dashboards & Visualizations
Highlighted

HTTP Event Collectors Invalid Token

Path Finder

I am having issue with multiple sets of HTTP Event Collectors we have running, each of which are throwing a "{"text":"Invalid token","code":4}" message, as shown below, when I ran a simple curl command against them.

[root@ip-10-0-17-167 ~]# curl -k  https://<<EC_URL>>:8088/services/collector/event -H "Authorization: Splunk 297B4C96-5B44-44D2-A9C1-873862AAD558" -d '{"event": "hello world"}'
{"text":"Invalid token","code":4}

This is happening with several tokens, all of which were previously working without issues. The only thing that has changed that I am aware of since I last tested the functionality (at build out) was a minor upgrade from v6.3.3 to v6.3.9. With that said, I have tested both existing (pre-upgrade) and new (post-upgrade) tokens, both with same result.

We are using a Deployment server to generate the tokens from within the UI and deploy them out to the HTTP Event Collectors. On the Deployment server, all of the tokens are listed under the splunk_httpinput app, including the one I am using in the curl command provided above.

[root@ip-10-0-16-52 splunk_httpinput]# cat /opt/splunk/etc/deployment-apps/splunk_httpinput/local/inputs.conf
[http]
disabled = 0
port = 8088
enableSSL = 1
dedicatedIoThreads = 2
maxThreads = 0
maxSockets = 0

...

[http://adslot-lambda]
disabled = 0
index = app
sourcetype = adslot-lambda
token = 297B4C96-5B44-44D2-A9C1-873862AAD558

I also confrimed that the tokens, including the one I am using in the curl command provided above, are deployed to the HTTP Event Collector I am pointed to. It is listed under the splunk_httpinput app just like it is listed on the Deployment server and Splunk has picked up the inputs setting following the reload.

[root@ip-10-0-18-38 apps]# cat /opt/splunk/etc/apps/splunk_httpinput/local/inputs.conf
[http]
disabled = 0
port = 8088
enableSSL = 1
dedicatedIoThreads = 2
maxThreads = 0
maxSockets = 0

...

[http://adslot-lambda]
disabled = 0
index = app
sourcetype = adslot-lambda
token = 297B4C96-5B44-44D2-A9C1-873862AAD558

[root@ip-10-0-18-38 apps]# /opt/splunk/bin/splunk cmd btool inputs --debug list

...

/opt/splunk/etc/apps/splunk_httpinput/local/inputs.conf                [http://adslot-lambda]
/opt/splunk/etc/system/default/inputs.conf                             _rcvbuf = 1572864
/opt/splunk/etc/apps/splunk_httpinput/local/inputs.conf                disabled = 0
/opt/splunk/etc/system/local/inputs.conf                               host = ip-10-0-18-38
/opt/splunk/etc/apps/splunk_httpinput/local/inputs.conf                index = app
/opt/splunk/etc/apps/splunk_httpinput/local/inputs.conf                sourcetype = adslot-lambda
/opt/splunk/etc/apps/splunk_httpinput/local/inputs.conf                token = 297B4C96-5B44-44D2-A9C1-873862AAD558

Please let me know if additional informaiton is needed and thanks in advanced for any assistance you can provide me.

0 Karma
Highlighted

Re: HTTP Event Collectors Invalid Token

SplunkTrust
SplunkTrust

Make sure useDeploymentServer is not in the config in your heavy forwarders.

View solution in original post

Highlighted

Re: HTTP Event Collectors Invalid Token

Path Finder

To make sure I did not screw anything up that I am not aware with our existing/broken Event Collectors, I started with a new setup and I am still having the same problems. I used the process outlined on Splunk Doc's website: http://docs.splunk.com/Documentation/Splunk/latest/Data/UsetheHTTPEventCollector

On the Deployment server I copied the "splunkhttpinput" app from the apps folder into the deployment-apps folder. From the Deployment server's UI, I enabled the EC service and checked the "Use Deployment Server" option. I also created a test token and it along with the settings are showing up in the "splunkhttpinput" app.

[root@ip-10-0-18-55 ~]# cat /opt/splunk/etc/deployment-apps/splunk_httpinput/default/inputs.conf
[http]
useDeploymentServer = 1
disabled = 0
port = 8088
enableSSL = 1
dedicatedIoThreads = 2
maxThreads = 0
maxSockets = 0

[root@ip-10-0-18-55 ~]# cat /opt/splunk/etc/deployment-apps/splunk_httpinput/local/inputs.conf
[http]
_rcvbuf = 1572864
allowSslCompression = true
allowSslRenegotiation = true
host = ip-10-0-18-55
index = app
sslVersions = *,-ssl2
enableSSL = 1

[http://Splunk EWE Admins]
disabled = 0
host = ip-10-0-18-55
index = app
sourcetype = splunkeweadmins
token = 04BD0B3D-A37C-4403-80E0-CDF37F5E9892

I setup the serverclass.conf file on the Deployment server to deploy the "splunk_httpinput" app, along with our outputs.conf app, to the one Event Collector. Both apps, along with the test token are now showing on the Event Collector.

[root@ip-10-0-18-122 ~]# cat /opt/splunk/etc/apps/splunk_httpinput/default/inputs.conf
[http]
useDeploymentServer = 1
disabled = 0
port = 8088
enableSSL = 1
dedicatedIoThreads = 2
maxThreads = 0
maxSockets = 0

[root@ip-10-0-18-122 ~]# cat /opt/splunk/etc/apps/splunk_httpinput/local/inputs.conf
[http]
_rcvbuf = 1572864
allowSslCompression = true
allowSslRenegotiation = true
host = ip-10-0-18-55
index = app
sslVersions = *,-ssl2
enableSSL = 1

[http://Splunk EWE Admins]
disabled = 0
host = ip-10-0-18-55
index = app
sourcetype = splunkeweadmins
token = 04BD0B3D-A37C-4403-80E0-CDF37F5E9892

However, when I send a test event using curl and my test token, I still get the ""Invalid token" error message and nothing indexed into the Splunk environment.

[root@ip-10-0-18-55 ~]# curl -k  https://<<ec_endpoint>>:8088/services/collector/event -H "Authorization: Splunk 04BD0B3D-A37C-4403-80E0-CDF37F5E9892" -d '{"event": "hello world"}'
{"text":"Invalid token","code":4}

Any other thoughts? Is there something wrong with my setup or configuration?

0 Karma
Highlighted

Re: HTTP Event Collectors Invalid Token

New Member

Any insights? I still have this issue. Thanks

0 Karma