How can I tell splunk to ignore the first few lines for TCP port indexing?

[tamduong16]

I opened a TCP port to capture incoming data. But I don't know what I could do to filter out the first 10 lines before my JSON syntax. I tried _json, log4j source type and also add Preamble regex to filter out the first 10 lines but it still being add to my index. What can I do?

**POST / HTTP/1.1
Host: xxxxxxxx
Accept: */*
User-Agent: python-requests/2.18.4
Content-Type: application/x-www-form-urlencoded
Content-Length: 11
Accept-Encoding: gzip, deflate
X-Forwarded-For: xxxxx
Via: xxxxx(McAfee Web Gateway xxxxx)
Connection: Keep-Alive
{
  "device":"sx20",
  "state":"Registered"
}**

[tamduong16 · Accepted solution]

I were able to filter them out by putting in a transform stanza. This post solved my problem. https://answers.splunk.com/answers/49366/how-to-ignore-first-three-line-of-my-log.html

[acharlieh]

It looks like you're using an HTTP POST to send JSON with the wrong Content-Type (it should be application/json if you were sending application/x-www-form-urlencoded the body would look like device=sx20&state=Registered ) to the TCP port, and the 10 lines that you want to get rid of are HTTP headers.

Why not setup and leverage the HTTP Event Collector instead? That way it properly handles the HTTP request, and you get your message indexed without them. You may need to do some manipulation either with the source python code, or possibly with the Gateway, but it seems like a good option to me.

Check out the Developer Site on the HTTP Event Collector: http://dev.splunk.com/view/event-collector/SP-CAAAE6M

[tamduong16]

@acharlieh - thank you for the answer. I really want to do an HTTP Event but this will be something entirely different. As I know this is a function in cisco api, it doesn't suppport token yet and I don't have the power to control it. Any idea?

POST / HTTP/1.1
Host: 1234.1341:514
Content-Type: application/json; charset=utf-8
Accept: text/, application/octet-stream, image/, audio/*
User-Agent: Cisco/CE
Accept-Charset: ISO-8859-1,utf-8
Content-Length: 681
{
"Status":{
"Identification":{
"SystemName":{
"Value":"xxx@video.cisco.com"
},
"MACAddress":{
"Value":"00:5f:6f:0f:cf:ff"
},
"IPAddress":{
"Value":"123.23.23"
},
"ProductType":{
"Value":"Cisco Codec"
},
"ProductID":{
"Value":"Cisco TelePresence SX20"
},
"SWVersion":{
"Value":"ce9.3.0.7394032fb9c"
},
"SerialNumber":{
"Value":"xxxxxN"
}
},
"SIP":{
"id":"",
"Registration":[
{
"id":"1",
"Status":{
"Value":"Deregister"
}
}
]
}
}
}

[acharlieh]

So it's an HTTP POST already... if you can't insert a token in from the source system, then setup a level 7 proxy/load balancer (F5 LTM, or Apache mod_proxy, or HAProxy, or others), to front your HTTP Event Collectors and insert the Authorization header with token as it's passed back to your HEC(s)

If you need more manipulation... you can develop an App server that does the transformation and pass it back...

[tamduong16]

Please excuse my ignorance. Where can I setup this load balancer. Could you give me a little more information? Thank you!

[sloshburch]

I think that moved the topic beyond the scope of Splunk and into infrastructure in general...