I am having issue with multiple sets of HTTP Event Collectors we have running, each of which are throwing a "{"text":"Invalid token","code":4}" message, as shown below, when I ran a simple curl command against them.
[root@ip-10-0-17-167 ~]# curl -k https://<<EC_URL>>:8088/services/collector/event -H "Authorization: Splunk 297B4C96-5B44-44D2-A9C1-873862AAD558" -d '{"event": "hello world"}'
{"text":"Invalid token","code":4}
This is happening with several tokens, all of which were previously working without issues. The only thing that has changed that I am aware of since I last tested the functionality (at build out) was a minor upgrade from v6.3.3 to v6.3.9. With that said, I have tested both existing (pre-upgrade) and new (post-upgrade) tokens, both with same result.
We are using a Deployment server to generate the tokens from within the UI and deploy them out to the HTTP Event Collectors. On the Deployment server, all of the tokens are listed under the splunk_httpinput app, including the one I am using in the curl command provided above.
[root@ip-10-0-16-52 splunk_httpinput]# cat /opt/splunk/etc/deployment-apps/splunk_httpinput/local/inputs.conf
[http]
disabled = 0
port = 8088
enableSSL = 1
dedicatedIoThreads = 2
maxThreads = 0
maxSockets = 0
...
[http://adslot-lambda]
disabled = 0
index = app
sourcetype = adslot-lambda
token = 297B4C96-5B44-44D2-A9C1-873862AAD558
I also confrimed that the tokens, including the one I am using in the curl command provided above, are deployed to the HTTP Event Collector I am pointed to. It is listed under the splunk_httpinput app just like it is listed on the Deployment server and Splunk has picked up the inputs setting following the reload.
[root@ip-10-0-18-38 apps]# cat /opt/splunk/etc/apps/splunk_httpinput/local/inputs.conf
[http]
disabled = 0
port = 8088
enableSSL = 1
dedicatedIoThreads = 2
maxThreads = 0
maxSockets = 0
...
[http://adslot-lambda]
disabled = 0
index = app
sourcetype = adslot-lambda
token = 297B4C96-5B44-44D2-A9C1-873862AAD558
[root@ip-10-0-18-38 apps]# /opt/splunk/bin/splunk cmd btool inputs --debug list
...
/opt/splunk/etc/apps/splunk_httpinput/local/inputs.conf [http://adslot-lambda]
/opt/splunk/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunk/etc/apps/splunk_httpinput/local/inputs.conf disabled = 0
/opt/splunk/etc/system/local/inputs.conf host = ip-10-0-18-38
/opt/splunk/etc/apps/splunk_httpinput/local/inputs.conf index = app
/opt/splunk/etc/apps/splunk_httpinput/local/inputs.conf sourcetype = adslot-lambda
/opt/splunk/etc/apps/splunk_httpinput/local/inputs.conf token = 297B4C96-5B44-44D2-A9C1-873862AAD558
Please let me know if additional informaiton is needed and thanks in advanced for any assistance you can provide me.
Make sure useDeploymentServer is not in the config in your heavy forwarders.
If you are using deploment server to create the token and push it to your heavy forwarders where it should be actually authenticate then you have to:
1. To make sure you change useDeploymentServer flag to true as below.
useDeploymentServer = 1
When this option is set to 1 and you make UI-based HEC changes on the deployment server, those changes are placed directly in the $SPLUNK_HOME/etc/deployment-apps/splunk_httpinput/ folder, rather than in $SPLUNK_HOME/etc/apps/folder.
Because if your inputs changes is there in the $SPLUNK_HOME/etc/apps/<anyapp>/inputs.conf on deployment server and also in your Heavy forwarder . Then the rest/curl call to token will end up in Invalid token response code 4.
Any insights? I still have this issue. Thanks
Make sure useDeploymentServer is not in the config in your heavy forwarders.
To make sure I did not screw anything up that I am not aware with our existing/broken Event Collectors, I started with a new setup and I am still having the same problems. I used the process outlined on Splunk Doc's website: http://docs.splunk.com/Documentation/Splunk/latest/Data/UsetheHTTPEventCollector
On the Deployment server I copied the "splunk_httpinput" app from the apps folder into the deployment-apps folder. From the Deployment server's UI, I enabled the EC service and checked the "Use Deployment Server" option. I also created a test token and it along with the settings are showing up in the "splunk_httpinput" app.
[root@ip-10-0-18-55 ~]# cat /opt/splunk/etc/deployment-apps/splunk_httpinput/default/inputs.conf
[http]
useDeploymentServer = 1
disabled = 0
port = 8088
enableSSL = 1
dedicatedIoThreads = 2
maxThreads = 0
maxSockets = 0
[root@ip-10-0-18-55 ~]# cat /opt/splunk/etc/deployment-apps/splunk_httpinput/local/inputs.conf
[http]
_rcvbuf = 1572864
allowSslCompression = true
allowSslRenegotiation = true
host = ip-10-0-18-55
index = app
sslVersions = *,-ssl2
enableSSL = 1
[http://Splunk EWE Admins]
disabled = 0
host = ip-10-0-18-55
index = app
sourcetype = splunkeweadmins
token = 04BD0B3D-A37C-4403-80E0-CDF37F5E9892
I setup the serverclass.conf file on the Deployment server to deploy the "splunk_httpinput" app, along with our outputs.conf app, to the one Event Collector. Both apps, along with the test token are now showing on the Event Collector.
[root@ip-10-0-18-122 ~]# cat /opt/splunk/etc/apps/splunk_httpinput/default/inputs.conf
[http]
useDeploymentServer = 1
disabled = 0
port = 8088
enableSSL = 1
dedicatedIoThreads = 2
maxThreads = 0
maxSockets = 0
[root@ip-10-0-18-122 ~]# cat /opt/splunk/etc/apps/splunk_httpinput/local/inputs.conf
[http]
_rcvbuf = 1572864
allowSslCompression = true
allowSslRenegotiation = true
host = ip-10-0-18-55
index = app
sslVersions = *,-ssl2
enableSSL = 1
[http://Splunk EWE Admins]
disabled = 0
host = ip-10-0-18-55
index = app
sourcetype = splunkeweadmins
token = 04BD0B3D-A37C-4403-80E0-CDF37F5E9892
However, when I send a test event using curl and my test token, I still get the ""Invalid token" error message and nothing indexed into the Splunk environment.
[root@ip-10-0-18-55 ~]# curl -k https://<<ec_endpoint>>:8088/services/collector/event -H "Authorization: Splunk 04BD0B3D-A37C-4403-80E0-CDF37F5E9892" -d '{"event": "hello world"}'
{"text":"Invalid token","code":4}
Any other thoughts? Is there something wrong with my setup or configuration?