We are Pushing the cloudflare logs to S3 and getting the logs to splunk with help of Splunk Add on AWS
Now we want to filter the logs before indexing only certain logs to allow to index please advise how do we achieve the same. if possible with example
Hi @kagamalai
can you share with me a sample log and what you want filter?
Regards
Alessandro
Sample log, i wan to to index only where the "WAFFlags":"0" events
{"BotScore":82,"BotScoreSrc":"Machine Learning","CacheCacheStatus":"unknown","CacheResponseBytes":2298,"CacheResponseStatus":200,"CacheTieredFill":false,"ClientASN":6128,"ClientCountry":"us","ClientDeviceType":"desktop","ClientIP":"69.117.161.105","ClientIPClass":"noRecord","ClientMTLSAuthCertFingerprint":"","ClientMTLSAuthStatus":"unknown","ClientRequestBytes":7939,"ClientRequestHost":"www.xxxxxxxx.com","ClientRequestMethod":"POST","ClientRequestPath":"/native/","ClientRequestProtocol":"HTTP/1.1","ClientRequestReferer":"","ClientRequestScheme":"https","ClientRequestSource":"eyeball","ClientRequestURI":"/native/?nativeApp=true","ClientRequestUserAgent":"xxxxxxxxRelease/113 CFNetwork/1220.1 Darwin/20.3.0","ClientSSLCipher":"ECDHE-RSA-AES128-GCM-SHA256","ClientSSLProtocol":"TLSv1.2","ClientSrcPort":59068,"ClientTCPRTTMs":20,"ClientXRequestedWith":"","EdgeCFConnectingO2O":false,"EdgeColoCode":"EWR","EdgeColoID":386,"EdgeEndTimestamp":"2021-06-02T10:04:49Z","EdgePathingOp":"wl","EdgePathingSrc":"macro","EdgePathingStatus":"nr","EdgeRateLimitAction":"","EdgeRateLimitID":0,"EdgeRequestHost":"www.xxxxxxxx.com","EdgeResponseBodyBytes":181,"EdgeResponseBytes":1266,"EdgeResponseCompressionRatio":1.1,"EdgeResponseContentType":"application/json","EdgeResponseStatus":200,"EdgeServerIP":"172.70.110.120","EdgeStartTimestamp":"2021-06-02T10:04:49Z","EdgeTimeToFirstByteMs":235,"FirewallMatchesActions":[],"FirewallMatchesRuleIDs":[],"FirewallMatchesSources":[],"OriginDNSResponseTimeMs":0,"OriginIP":"50.204.152.175","OriginRequestHeaderSendDurationMs":0,"OriginResponseBytes":0,"OriginResponseDurationMs":144,"OriginResponseHTTPExpires":"","OriginResponseHTTPLastModified":"","OriginResponseHeaderReceiveDurationMs":144,"OriginResponseStatus":200,"OriginResponseTime":144000000,"OriginSSLProtocol":"TLSv1.2","OriginTCPHandshakeDurationMs":0,"OriginTLSHandshakeDurationMs":0,"ParentRayID":"00","RayID":"658fdc1918b3183d","SecurityLevel":"med","SmartRouteColoID":0,"UpperTierColoID":0,"WAFAction":"unknown","WAFFlags":"0","WAFMatchedVar":"","WAFProfile":"unknown","WAFRuleID":"","WAFRuleMessage":"","WorkerCPUTime":0,"WorkerStatus":"unknown","WorkerSubrequest":false,"WorkerSubrequestCount":0,"ZoneID":12628270,"ZoneName":"xxxxxxxx.com"}
Hi @kagamalai my friend
how are you? sorry for the delay but was really a busy period
you can filter the log config the props and transforms file on your indexer or you HF, depends about your architecture.
https://docs.splunk.com/Documentation/Splunk/8.2.0/Forwarding/Routeandfilterdatad
anyway the configuration is:
props.conf
[<your sourcetype>]
TRANSFORMS-set= setnull,setparsing
transfomrs.conf
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue
[setparsing]
REGEX = \"WAFFlags\":\"0\"
DEST_KEY = queue
FORMAT = indexQueue
Thanks for your reply
But I am getting below source format from S3 like below sample file name every 30 min ( random file name)
index="cloudflare" source source="s3://sgd-prod-cloudflare/logs/XXXXXX.net/20210614/20210614t224909z_20210614t224939z_65fd8f67.log.gz"
index="cloudflare" source source="s3://sgd-prod-cloudflare/logs/XXXXXX.net/20210614/20210614t034936z_20210614t035006z_e007437b.log.gz"
index="cloudflare" source source="s3://sgd-prod-cloudflare/logs/XXXXXX.net/20210614/20210614t225009z_20210614t225039z_4d1f18af.log.gz"
Please advise how to do this ? I am waiting for your reply
Hi @kagamalai
please provide me the sourcetype name not the source ok?
I could give you the correct conf file configuration
Regards
Ale
Hi,
Below is the source type
index="cloudflare" sourcetype="cloudflare:json"
thanks
Hi @kagamalai
please create this two conf file, I suggest to create a small app and put these files on the local folder on the indexer
props.conf
[cloudflare:json]
TRANSFORMS-set= setnull,setparsing
transfomrs.conf
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue
[setparsing]
REGEX = \"WAFFlags\":\"0\"
DEST_KEY = queue
FORMAT = indexQueue
please read the documentation
Thank you let me try this.