REGEX filtering http headers to null queue

beaunewcomb — Thu, 16 Aug 2012 15:09:41 GMT

Trying to strip the header out of an incoming http post and only index the json for parsing with spath. Having an issue with the regex I believe.

props.conf:

[akamai_post_json]
TRANSFORMS-t1=nullqueue_httpheader

transforms.conf:

[nullqueue_httpheader]
REGEX=^(?:POST|User-Agent|Accept|Content-Length|Expect|Content-Type|Content-Disposition)gm
DEST_KEY=queue
FORMAT=nullQueue

Here's a sample event:

POST / HTTP/1.1
User-Agent: curl/7.26.0
Host: localhost
Accept: */*
Content-Length: 2552
Expect: 100-continue
Content-Type: multipart/form-data; boundary=----------------------------0b1c32056fc5
------------------------------0b1c32056fc5
Content-Disposition: form-data; name="fileupload"; filename="default_connector_schema_1.0.json"
Content-Type: application/octet-stream
{
  "apiType" : "String",
  "apiFormat" : "String",
  "apiVersion" : 0,
  "ID" : "String",
  "startTime" : "String",
  "eventType" : "String",
  "cpCode" : 0,
  "message" : {
    "protocol" : "0",
    "protoVersion" : 0,
    "clientIP" : "String",
    "reqPort" : 0,
    "reqHost" : "String",
    "reqMethod" : "String",
    "reqPath" : "String",
    "reqQuery" : "String",
    "reqContType" : "String",
    "reqContLen" : 0,
    "sslProtocol" : "String",
    "sslVersion" : 0,
    "respStatus" : 0,
    "respRedirURL" : "String",
    "respContType" : "String",
    "respContLen" : 0,
    "respBytesServed" : 0,
    "userAgent" : "String",
    "originHostname" : "String"
  },
  "httpHeaders" : {
    "reqHeader" : {
      "accEnc" : "String",
      "accLang" : "String",
      "auth" : "String",
      "cacheCtl" : "String",
      "connection" : "String",
      "contMD5" : "String",
      "cookie" : "String",
      "DNT" : "String",
      "ifMatch" : "String",
      "ifMod" : "String",
      "ifNoMatch" : "String",
      "pragma" : "String",
      "range" : "String",
      "referer" : "String",
      "TE" : "String",
      "upgrade" : "String",
      "via" : "String",
      "xFrwdFor" : "String",
      "xReqWith" : "String"
    },
    "respHeader" : {
      "cacheCtl" : "String",
      "connection" : "String",
      "contEnc" : "String",
      "contLang" : "String",
      "contLen" : "String",
      "contMD5" : "String",
      "contDisp" : "String",
      "contRange" : "String",
      "date" : "String",
      "eTag" : "String",
      "expires" : "String",
      "lastMod" : "String",
      "p3p" : "String",
      "pragma" : "String",
      "server" : "String",
      "setCookie" : "String",
      "trailer" : "String",
      "transEnc" : "String",
      "vary" : "String",
      "warning" : "String",
      "wwwAuth" : "String"
    }
  },
  "performance" : {
    "reqHeadSize" : 0,
    "reqBodySize" : 0,
    "respHeadSize" : 0,
    "respBodySize" : "String",
    "downloadTime" : "String",
    "originName" : "String",
    "originIP" : "String",
    "originInitIP" : "String",
    "originRetry" : 0,
    "lastMileRTT" : 0,
    "lastMileBW" : 0,
    "netOriginRTT" : 0,
    "cacheStatus" : "String",
    "lastByte" : true,
    "cliCountry" : "String",
    "edgeIP" : "String",
    "reqID" : "String"
  }
}
------------------------------0b1c32056fc5--

Re: REGEX filtering http headers to null queue

kristian_kolb — Thu, 16 Aug 2012 15:28:28 GMT

I'm afraid that your regex would never do what you want it to. Sending stuff to the nullQueue works on a per-event basis, not on separate lines in a multiline event.

You're probably better off looking at the SEDCMD-name in props.conf. In the docs there is an example of masking credit card numbers and other sensitive data, but you could easily filter out the unwanted lines with a sed script.

EDIT: typo
/k

Re: REGEX filtering http headers to null queue

beaunewcomb — Mon, 28 Sep 2020 12:16:52 GMT

So instead of the transform in props.conf, I use SEDCMD, but I'm still not seeing ANY change. Is the below correct for remove the POST line?

props.conf:

[akamai_post_json]
SEDCMD-httpheader = s/(?mg)^POST.*$\n//g

topic Re: REGEX filtering http headers to null queue in Getting Data In

REGEX filtering http headers to null queue

Re: REGEX filtering http headers to null queue

Re: REGEX filtering http headers to null queue