Reporting

How do I filter the incoming logs from S3 bucket

kagamalai
Explorer

We are Pushing the cloudflare logs to S3 and getting the logs to splunk with help of Splunk Add on AWS 

Now we want to filter the logs before indexing only certain logs to allow to index  please advise how do we achieve the same. if possible with example 

Labels (1)
0 Karma

aasabatini
Motivator

Hi @kagamalai 

can you share with me a sample log and what you want filter?

Regards

Alessandro

“The answer is out there, Neo, and it’s looking for you, and it will find you if you want it to.”
0 Karma

kagamalai
Explorer

Sample log, i wan to to index only where the "WAFFlags":"0" events

 

{"BotScore":82,"BotScoreSrc":"Machine Learning","CacheCacheStatus":"unknown","CacheResponseBytes":2298,"CacheResponseStatus":200,"CacheTieredFill":false,"ClientASN":6128,"ClientCountry":"us","ClientDeviceType":"desktop","ClientIP":"69.117.161.105","ClientIPClass":"noRecord","ClientMTLSAuthCertFingerprint":"","ClientMTLSAuthStatus":"unknown","ClientRequestBytes":7939,"ClientRequestHost":"www.xxxxxxxx.com","ClientRequestMethod":"POST","ClientRequestPath":"/native/","ClientRequestProtocol":"HTTP/1.1","ClientRequestReferer":"","ClientRequestScheme":"https","ClientRequestSource":"eyeball","ClientRequestURI":"/native/?nativeApp=true","ClientRequestUserAgent":"xxxxxxxxRelease/113 CFNetwork/1220.1 Darwin/20.3.0","ClientSSLCipher":"ECDHE-RSA-AES128-GCM-SHA256","ClientSSLProtocol":"TLSv1.2","ClientSrcPort":59068,"ClientTCPRTTMs":20,"ClientXRequestedWith":"","EdgeCFConnectingO2O":false,"EdgeColoCode":"EWR","EdgeColoID":386,"EdgeEndTimestamp":"2021-06-02T10:04:49Z","EdgePathingOp":"wl","EdgePathingSrc":"macro","EdgePathingStatus":"nr","EdgeRateLimitAction":"","EdgeRateLimitID":0,"EdgeRequestHost":"www.xxxxxxxx.com","EdgeResponseBodyBytes":181,"EdgeResponseBytes":1266,"EdgeResponseCompressionRatio":1.1,"EdgeResponseContentType":"application/json","EdgeResponseStatus":200,"EdgeServerIP":"172.70.110.120","EdgeStartTimestamp":"2021-06-02T10:04:49Z","EdgeTimeToFirstByteMs":235,"FirewallMatchesActions":[],"FirewallMatchesRuleIDs":[],"FirewallMatchesSources":[],"OriginDNSResponseTimeMs":0,"OriginIP":"50.204.152.175","OriginRequestHeaderSendDurationMs":0,"OriginResponseBytes":0,"OriginResponseDurationMs":144,"OriginResponseHTTPExpires":"","OriginResponseHTTPLastModified":"","OriginResponseHeaderReceiveDurationMs":144,"OriginResponseStatus":200,"OriginResponseTime":144000000,"OriginSSLProtocol":"TLSv1.2","OriginTCPHandshakeDurationMs":0,"OriginTLSHandshakeDurationMs":0,"ParentRayID":"00","RayID":"658fdc1918b3183d","SecurityLevel":"med","SmartRouteColoID":0,"UpperTierColoID":0,"WAFAction":"unknown","WAFFlags":"0","WAFMatchedVar":"","WAFProfile":"unknown","WAFRuleID":"","WAFRuleMessage":"","WorkerCPUTime":0,"WorkerStatus":"unknown","WorkerSubrequest":false,"WorkerSubrequestCount":0,"ZoneID":12628270,"ZoneName":"xxxxxxxx.com"}

0 Karma

kagamalai
Explorer

@aasabatini,

Any Luck ?

 

0 Karma

aasabatini
Motivator

Hi @kagamalai  my friend

how are you? sorry for the delay but was really a busy period

you can filter the log config the props and transforms file on your indexer or you HF, depends about your architecture.

https://docs.splunk.com/Documentation/Splunk/8.2.0/Forwarding/Routeandfilterdatad

anyway the configuration is:
props.conf

[<your sourcetype>]
TRANSFORMS-set= setnull,setparsing

transfomrs.conf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = \"WAFFlags\":\"0\"
DEST_KEY = queue
FORMAT = indexQueue

 

“The answer is out there, Neo, and it’s looking for you, and it will find you if you want it to.”
0 Karma

kagamalai
Explorer

Thanks for your reply 

But I am getting below source format from S3 like below sample file name every 30 min ( random file name)

index="cloudflare" source source="s3://sgd-prod-cloudflare/logs/XXXXXX.net/20210614/20210614t224909z_20210614t224939z_65fd8f67.log.gz"
index="cloudflare" source source="s3://sgd-prod-cloudflare/logs/XXXXXX.net/20210614/20210614t034936z_20210614t035006z_e007437b.log.gz"
index="cloudflare" source source="s3://sgd-prod-cloudflare/logs/XXXXXX.net/20210614/20210614t225009z_20210614t225039z_4d1f18af.log.gz"

Please advise how to do this ? I am waiting for your reply 

0 Karma

aasabatini
Motivator

Hi @kagamalai 

please provide me the sourcetype name not the source ok?

I could give you the correct conf file configuration

Regards

Ale

“The answer is out there, Neo, and it’s looking for you, and it will find you if you want it to.”
0 Karma

kagamalai
Explorer

Hi,

Below is the source type 

index="cloudflare" sourcetype="cloudflare:json"

thanks 

 

0 Karma

aasabatini
Motivator

Hi @kagamalai 

please create this two conf file,  I suggest to create a small app and put these files on the local folder on the indexer

props.conf

[cloudflare:json]
TRANSFORMS-set= setnull,setparsing

transfomrs.conf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = \"WAFFlags\":\"0\"
DEST_KEY = queue
FORMAT = indexQueue

 

please read the documentation

 

“The answer is out there, Neo, and it’s looking for you, and it will find you if you want it to.”
0 Karma

kagamalai
Explorer

Thank you let me try this.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...