Getting Data In

How can I tell splunk to ignore the first few lines for TCP port indexing?

tamduong16
Contributor

I opened a TCP port to capture incoming data. But I don't know what I could do to filter out the first 10 lines before my JSON syntax. I tried _json, log4j source type and also add Preamble regex to filter out the first 10 lines but it still being add to my index. What can I do?

**POST / HTTP/1.1
Host: xxxxxxxx
Accept: */*
User-Agent: python-requests/2.18.4
Content-Type: application/x-www-form-urlencoded
Content-Length: 11
Accept-Encoding: gzip, deflate
X-Forwarded-For: xxxxx
Via: xxxxx(McAfee Web Gateway xxxxx)
Connection: Keep-Alive
{
  "device":"sx20",
  "state":"Registered"
}**
0 Karma
1 Solution

tamduong16
Contributor

I were able to filter them out by putting in a transform stanza. This post solved my problem. https://answers.splunk.com/answers/49366/how-to-ignore-first-three-line-of-my-log.html

View solution in original post

0 Karma

tamduong16
Contributor

I were able to filter them out by putting in a transform stanza. This post solved my problem. https://answers.splunk.com/answers/49366/how-to-ignore-first-three-line-of-my-log.html

0 Karma

acharlieh
Influencer

It looks like you're using an HTTP POST to send JSON with the wrong Content-Type (it should be application/json if you were sending application/x-www-form-urlencoded the body would look like device=sx20&state=Registered ) to the TCP port, and the 10 lines that you want to get rid of are HTTP headers.

Why not setup and leverage the HTTP Event Collector instead? That way it properly handles the HTTP request, and you get your message indexed without them. You may need to do some manipulation either with the source python code, or possibly with the Gateway, but it seems like a good option to me.

Check out the Developer Site on the HTTP Event Collector: http://dev.splunk.com/view/event-collector/SP-CAAAE6M

tamduong16
Contributor

@acharlieh - thank you for the answer. I really want to do an HTTP Event but this will be something entirely different. As I know this is a function in cisco api, it doesn't suppport token yet and I don't have the power to control it. Any idea?

POST / HTTP/1.1
Host: 1234.1341:514
Content-Type: application/json; charset=utf-8
Accept: text/, application/octet-stream, image/, audio/*
User-Agent: Cisco/CE
Accept-Charset: ISO-8859-1,utf-8
Content-Length: 681
{
"Status":{
"Identification":{
"SystemName":{
"Value":"xxx@video.cisco.com"
},
"MACAddress":{
"Value":"00:5f:6f:0f:cf:ff"
},
"IPAddress":{
"Value":"123.23.23"
},
"ProductType":{
"Value":"Cisco Codec"
},
"ProductID":{
"Value":"Cisco TelePresence SX20"
},
"SWVersion":{
"Value":"ce9.3.0.7394032fb9c"
},
"SerialNumber":{
"Value":"xxxxxN"
}
},
"SIP":{
"id":"",
"Registration":[
{
"id":"1",
"Status":{
"Value":"Deregister"
}
}
]
}
}
}

0 Karma

acharlieh
Influencer

So it's an HTTP POST already... if you can't insert a token in from the source system, then setup a level 7 proxy/load balancer (F5 LTM, or Apache mod_proxy, or HAProxy, or others), to front your HTTP Event Collectors and insert the Authorization header with token as it's passed back to your HEC(s)

If you need more manipulation... you can develop an App server that does the transformation and pass it back...

0 Karma

tamduong16
Contributor

Please excuse my ignorance. Where can I setup this load balancer. Could you give me a little more information? Thank you!

0 Karma

sloshburch
Splunk Employee
Splunk Employee

I think that moved the topic beyond the scope of Splunk and into infrastructure in general...

0 Karma
Get Updates on the Splunk Community!

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...

Reminder! Splunk Love Promo: $25 Visa Gift Card for Your Honest SOAR Review With ...

We recently launched our first Splunk Love Special, and it's gone phenomenally well, so we're doing it again, ...