Getting Data In
Highlighted

HTTP Event Collector: Getting error "HttpInputDataHandler - Parsing error".

I have Splunk set up as an HTTP Event Collector receiver and am seeing parsing errors in splunkd.log like: ERROR HttpInputDataHandler - Parsing error.

How do I resolve these?

Labels (1)
Highlighted

Re: HTTP Event Collector: Getting error "HttpInputDataHandler - Parsing error".

You may see Parsing errors similar to the ones below , however the event does not show the client ip.

03-17-2020 12:55:50.841 -0400 ERROR HttpInputDataHandler - Parsing error : Got unexpected null element while expecting event's raw text, totalRequestSize=133

05-29-2020 12:36:34.333 -0400 ERROR HttpInputDataHandler - Parsing error : Event field cannot be blank

05-29-2020 12:35:32.005 -0400 ERROR HttpInputDataHandler - Parsing error : While expecting event's raw text: Unexpected character while looking for value: '}', totalRequestSize=40

05-29-2020 12:33:03.569 -0400 ERROR HttpInputDataHandler - Parsing error : While expecting event object key: Unexpected character: 'e', totalRequestSize=66

You should group by the timestamp of the event as there will be 2 events logged (1 logging the parsing error and the other logging the response sent back to the client , which includes the client ip and the reply code).

For example:
05-29-2020 12:39:42.473 -0400 ERROR HttpInputDataHandler - Parsing error : No data
05-29-2020 12:39:42.473 -0400 ERROR HttpInputDataHandler - Failed processing http input, token name=idxclustertoken, channel=FE0ECFAD-13D5-401A-847D-77833DD77131, sourceIP=10.140.49.235, reply=5, eventsprocessed=0, httpinputbody_size=54

05-29-2020 13:00:10.016 -0400 ERROR HttpInputDataHandler - Parsing error : While expecting event object to start: Unexpected character while looking for value: '\', totalRequestSize=69
05-29-2020 13:00:10.016 -0400 ERROR HttpInputDataHandler - Failed processing http input, token name=idxclustertoken, channel=FE0ECFAD-13D5-401A-847D-77833DD77131, sourceIP=10.140.49.235, reply=6, eventsprocessed=1, httpinputbody_size=69

grouping by _time and host will make these parsing errors easier to decipher:

index=_internal source=*splunkd.log HttpInputDataHandler ERROR | stats values(_raw) by _time host

Based on the reply code and client source_IP, you should examine the syntax of the request sent by that client or check the health of the HEC receiver.

HEC reply codes:
reply HttpInputReply status eventmessage
0 Success OK Success
1 TokenDisabled FORBIDDEN Token disabled
2 NoAuthorization UNAUTHORIZED Token is required
3 InvalidAuthorization UNAUTHORIZED Invalid authorization
4 TokenNotFound FORBIDDEN Invalid token
5 NoData BAD
REQUEST No data
6 InvalidData BADREQUEST Invalid data format
7 IncorrectIndex BAD
REQUEST Incorrect index
8 ServerError has been removed as it is not used anywhere
9 ServerBusy SERVICEUNAVAILABLE Server is busy
10 NoChannel BAD
REQUEST Data channel is missing
11 InvalidChannel BADREQUEST Invalid data channel
12 NoEvent BAD
REQUEST Event field is required
13 BlankEvent BADREQUEST Event field cannot be blank
14 AckDisabled BAD
REQUEST ACK is disabled
15 UnsupportedType BADREQUEST Error in handling indexed fields
16 QueryStringAuthNotEnabled BAD
REQUEST Query string authorization is not enabled
17 HECHealthy OK HEC is healthy
18 QueuesFull SERVICEUNAVAILABLE HEC is unhealthy, queues are full
19 AckUnavailable SERVICE
UNAVAILABLE HEC is unhealthy, ack service unavailable
20 QueuesFullAckUnavailable SERVICE_UNAVAILABLE Hec is unhealthy, queues are full, ack service unavailable

Note: My test was on Splunk 8.0.4 where the "response" event which includes the sourceIP and reply fields are logged as loglevel=ERROR

earlier versions of Splunk require setting the HttpInputDataHandler component into DEBUG to see these events:

ie: set on the HEC receiver Splunk instance:
./splunk set log-level HttpInputDataHandler -level DEBUG

then back to normal:
./splunk set log-level HttpInputDataHandler -level WARN

View solution in original post

0 Karma
Highlighted

Re: HTTP Event Collector: Getting error "HttpInputDataHandler - Parsing error".

or you could also run a search that is easier to digest:

index=_internal source=*splunkd.log* log_level=ERROR OR log_level=DEBUG component=HttpInputDataHandler reply=* | eval response_to_client=case(reply=="0","success",reply=="1","Token disabled",reply=="2","Token is required",reply=="3","Invalid authorization",reply=="4","Invalid token",reply=="5","No data",reply=="6","Invalid data format",reply=="7","Incorrect index",reply=="9","Server is busy",reply=="10","Data channel is missing",reply=="11","Invalid data channel",reply=="12","Event field is required",reply=="13","Event field cannot be blank",reply=="14","ACK is disabled",reply=="15","Error in handling indexed fields",reply=="16","Query string authorization is not enabled",reply=="17","HEC is healthy",reply=="18","HEC is unhealthy, queues are full",reply=="19","HEC is unhealthy, ack service unavailable",reply=="20","Hec is unhealthy, queues are full, ack service unavailable") | stats count by host name channel source_IP response_to_client reply | rename host as "HEC Receiver" source_IP as "HEC client"
0 Karma
Speak Up for Splunk Careers!

We want to better understand the impact Splunk experience and expertise has has on individuals' careers, and help highlight the growing demand for Splunk skills.