Security

How to view events from HTTP Event Connector?

rajpraba
Explorer

I have installed AWS Splunk enterprise. We plan to redirect our Node JS application logs to Splunk. I have setup http event connector and got a token. I am able to send event logs from console (curl) as well as from application. But I dont see it anywhere.

The search/dashboard shows 0 events. But the Index says 1 MB...

curl -k https://54.xx.xx.xx:8088/services/collector -H 'Authorization: Splunk xxxxx  -d '{"source": "test-token-1", "sourcetype": "_json", "index":"sample", "event":"Hello, World!"}' 
{"text":"Success","code":0}

The name of the input token is 'test-token-1'. this is configured as source type _json and default index as 'sample'>

The search shows 0 events

I ssh to the box and see the folder

[ec2-user@ip-172-31-36-245 ~]$ sudo find / -name sample -type d
/opt/splunk/var/lib/splunk/sample

but don't have permission to see the content.

Any help is approeciated.

thanks
Raj

0 Karma
1 Solution

rajpraba
Explorer

Since, The response from console was
{"text":"Success","code":0}, It appears the event is logged.
After playing with this all day, I got a message,
'Disk Monitor: Cannot write data to index path '/opt/splunk/var/lib/splunk/_internaldb/db' because you are low on disk space on partition '/'. Indexing has been paused. Free disk space above 5000MB to resume indexing.'

However the search is showing 0. So, I would think the admin does not have rights to see any of the events logged. I also connected my Node JS app server logs. They seem be to logging but my "search" is empty. shows waiting for data. And shows 0 events in http event conenctor dahsboard

View solution in original post

rajpraba
Explorer

Since, The response from console was
{"text":"Success","code":0}, It appears the event is logged.
After playing with this all day, I got a message,
'Disk Monitor: Cannot write data to index path '/opt/splunk/var/lib/splunk/_internaldb/db' because you are low on disk space on partition '/'. Indexing has been paused. Free disk space above 5000MB to resume indexing.'

However the search is showing 0. So, I would think the admin does not have rights to see any of the events logged. I also connected my Node JS app server logs. They seem be to logging but my "search" is empty. shows waiting for data. And shows 0 events in http event conenctor dahsboard

starcher
Influencer

Sounds like you answered it. Indexing paused due to space.

0 Karma

rajpraba
Explorer

starcher:
Thank you so much for leading me into the solution. The aws micro medium instance I used which quickly filled with logs. I used small instance to play with before committing to c3.xlarge instance which works flawlessly.

Thanks

0 Karma

rajpraba
Explorer

Starcher:

Thanks. Let me answer what I had done. If I need to do something else for the steps please let me know.

  1. I see the Index named 'sample' in Setting->Indexes, I see a row for sample ($SPLUNK_DB/sample/db, 0 events, enabled)
  2. Settings->Http Event Collector->test-token-1 edit->Default Index ->Sample. (hope this is what is intended configuration)
  3. I am logged in as admin. I add all the roles available. Settings->Access Control -> Admin

Thanks

0 Karma

starcher
Influencer

Are you:
1. sure index = sample exists?
2. the HEC token was given permission to the index?
3. that you have permission to search index = sample if 1 and 2 above are true.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...

New in Observability Cloud - Explicit Bucket Histograms

Splunk introduces native support for histograms as a metric data type within Observability Cloud with Explicit ...