Splunk Search

How to filter IIS Logs with 80 and 443?

anandhalagaras1
Communicator

Hi Team,

We have a requirement to filter out the events from the IIS logs if the event contains ""GET / - 80 -" OR "GET / - 443 -" in the logs before the ingestion. 

QUERY:

index="*"  "GET / - 80 -" OR "GET / - 443 -"

Sample Format:

2022-12-12 00:38:10 xx.yyy.zzz.aaa GET / - 80 - xx.yyy.z.a - - x00 0 0 0

2022-12-12 00:44:45 xx.yyy.zzz.aaa GET / - 443 - xx.yyy.z.a - - y00 0 0 x

 

Hence kindly help with the relevant props and transforms for the same.

Labels (2)
0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi @anandhalagaras1,

maybe i misunderstood your requirement: do you want to take or discard events with 80 and 443?

if you want to take them and discard the others you can use my first solution:

in props.conf

[your_sourcetype]
TRANSFORMS-set= setnull,setparsing

in transormsconf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX =  GET\s*\/\s*-\s*80|443
DEST_KEY = queue
FORMAT = indexQueue

If you want to discard events with 80 and 443 you can use your solution.

Anyway, I don't like to use two stanzas, please try using only one, also because the sequence of commands (in props is relevant, so to avoid problems I prefer to have a single stanza.

Ciao.

Giuseppe

View solution in original post

gcusello
SplunkTrust
SplunkTrust

Hi @anandhalagaras1,

let me understand: do you want to filter events before indexing (index time) or during searches (search time)?

in the first case you lose all the other logs and you cannot use them for other searches (e.g. finding errors).

In this case you can follow the instructions at https://docs.splunk.com/Documentation/Splunk/9.0.3/Forwarding/Routeandfilterdatad#Filter_event_data_...

in few words, in props.conf:

[your_sourcetype]
TRANSFORMS-set= setnull,setparsing

in transforms.conf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX =  GET\s*\/\s*-\s*80|443
DEST_KEY = queue
FORMAT = indexQueue

if you want to filter events at search time, in other words only in displaying data, your search is almost correct: I'd use parenthesis:

index="*"  ("GET / - 80 -" OR "GET / - 443 -")

Ciao.

Giuseppe

anandhalagaras1
Communicator

@gcusello ,

Thanks for your response.

Actually my query was not to ingest the logs if GET / - 80 or GET / - 443 is present in the IIS logs.

i.e. We want to filter (ignore) the events before indexing (index time) .

And we are already ingesting the IIS logs into Splunk from the servers but if the below event is present in the logs for the sourcetype "xyz" then it needs to be filtered out i.e. it is not required to be ingested into Splunk itself.

ie GET / - 80  or GET / - 443 is present in the IIS logs then it is not required to be ingested into Splunk. 

2022-12-12 00:38:10 xx.yyy.zzz.aaa GET / - 80 - xx.yyy.z.a - - x00 0 0 0

2022-12-12 00:44:45 xx.yyy.zzz.aaa GET / - 443 - xx.yyy.z.a - - y00 0 0 x

Hence kindly help with the props and transforms for the same.

 

 

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @anandhalagaras1,

as you can see in my previous answer, you have to put the props.conf and transfroms.conf in the Indexers or (if present) on intermediate Heavy Forwarders, not on Universal Forwarders.

in the props.conf you have to indicate your sourcetype and add the row of TRANSFORMATION and in the transforms.conf you have to insert the rows I shared using the correct regex.

The one I shared should be correct but, test it.

ciao.

Giuseppe

0 Karma

anandhalagaras1
Communicator

@gcusello , I used the below props and transforms in my Heavy Forwarder server to eliminate the logs if it contains "GET / - 80 -"  or "GET / - 443 -"  as you have provided but still i can see the events with GET / - 80 - and GET / - 443 - are still getting ingested into Splunk. 

So anything needs to be changed in props and transforms. Kindly let me know.

props.conf:
[sourcetype]
TRANSFORMS-set= setnull,setparsing

transforms.conf:
[setnull]
REGEX = GET\s*\/\s*-\s*80|443
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = .
DEST_KEY = queue
FORMAT = indexQueue

 

Also I have tried this props and transforms as well but still the same.

[sourcetype]
TRANSFORMS-null= setnull152,setnull153


[setnull152]
REGEX = GET\s*\/\s*-\s*80
DEST_KEY = queue
FORMAT = nullQueue

[setnull153]
REGEX = GET\s*\/\s*-\s*443
DEST_KEY = queue
FORMAT = nullQueue

But still its not working. Can you kindly help me.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @anandhalagaras1,

at first check the regexes in Splunk Search using the regex command.

Then One question: before the Heavy Forwarder where are located the conf files, are there other Heavy Forwarders?

I suppose that in the props.conf stanza header, you inserte the correct sourcetype value of the data source to filter.

Ciao.

Giuseppe

0 Karma

anandhalagaras1
Communicator

@gcusello 

at first check the regexes in Splunk Search using the regex command. --> Yes I have checked in the Search Head and it seems to be fine.

index="*" sourcetype=**
| regex GET\s*\/\s*-\s*80

 

Then One question: before the Heavy Forwarder where are located the conf files, are there other Heavy Forwarders? --> The client machines forwarder output has been send to Heavy Forwarder 1 and from there it will reach the indexers so similarly I have placed the config files  props and transforms in an app  to the same Heavy Forwarder 1 server and also have restarted the services of the client machine as well but still the logs which contains 80 and 443 is not getting removed before ingestion.

I suppose that in the props.conf stanza header, you inserted the correct sourcetype value of the data source to filter.--> Yes exactly.

Currently I am using this props and transforms so is this a correct one to filter out the logs?

props.conf
[sourcetype]
TRANSFORMS-null= setnull152,setnull153

transforms.conf
[setnull152]
REGEX = GET\s*\/\s*-\s*80
DEST_KEY = queue
FORMAT = nullQueue

[setnull153]
REGEX = GET\s*\/\s*-\s*443
DEST_KEY = queue
FORMAT = nullQueue

0 Karma

anandhalagaras1
Communicator

@gcusello ,

Thank you.

This solution worked for sometime. But now the issue is the logs are coming with "get" (In lower case) and the filtration applied for "GET" (Upper case) so the condition is that if the logs are coming with 80 or 443 with "get" OR "GET" the logs should not be ingested into Splunk. So kindly provide the required regex. 

 

Sample Logs:

2023-01-29 02:38:37 xx.yyy.zzz.aaa get / - 80 - qq.www.e.rrr - - abc d e f

2023-01-29 02:38:37 xx.yyy.zzz.aaa GET/ - 80 - qq.www.e.rrr - - abc d e f

Currently used props and transforms for reference.

props:

[sourcetype]
TRANSFORMS-null= setnull1002
TZ = Etc/GMT

transforms:

[setnull1002]
REGEX = GET\s*\/\s*-\s*80|443
DEST_KEY = queue
FORMAT = nullQueue

Tags (1)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @anandhalagaras1,

you have to adapt the regex to this new requirement:

REGEX =  (GET|get)\s*\/\s*-\s*80|443

if one answer solves your need, please accept one answer for the other people of Community or tell me how I can help you.

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated 😉

0 Karma

anandhalagaras1
Communicator

@gcusello , When i use this query to check in Splunk search head i am getting an error as "

Unknown search command 'get'.
 

index="xxx" | regex (GET|get)\s*\/\s*-\s*80|443

 

So kindly help to check and update.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @anandhalagaras1,

this regex is for the transforms.conf file,

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX =  (GET|get)\s*\/\s*-\s*80|443
DEST_KEY = queue
FORMAT = indexQueue

if you use it in a search you have to use quotes:

index="xxx" 
| regex "(GET|get)\s*\/\s*-\s*80|443"

 Ciao.

Giuseppe

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @anandhalagaras1,

maybe i misunderstood your requirement: do you want to take or discard events with 80 and 443?

if you want to take them and discard the others you can use my first solution:

in props.conf

[your_sourcetype]
TRANSFORMS-set= setnull,setparsing

in transormsconf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX =  GET\s*\/\s*-\s*80|443
DEST_KEY = queue
FORMAT = indexQueue

If you want to discard events with 80 and 443 you can use your solution.

Anyway, I don't like to use two stanzas, please try using only one, also because the sequence of commands (in props is relevant, so to avoid problems I prefer to have a single stanza.

Ciao.

Giuseppe

Get Updates on the Splunk Community!

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...

Edge Processor Scaling, Energy & Manufacturing Use Cases, and More New Articles on ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...