How to filter IIS Logs with 80 and 443?

[anandhalagaras1]

Hi Team,

We have a requirement to filter out the events from the IIS logs if the event contains ""GET / - 80 -" OR "GET / - 443 -" in the logs before the ingestion. 

QUERY:

index="*"  "GET / - 80 -" OR "GET / - 443 -"

Sample Format:

2022-12-12 00:38:10 xx.yyy.zzz.aaa GET / - 80 - xx.yyy.z.a - - x00 0 0 0

2022-12-12 00:44:45 xx.yyy.zzz.aaa GET / - 443 - xx.yyy.z.a - - y00 0 0 x

Hence kindly help with the relevant props and transforms for the same.

[gcusello]

Hi @anandhalagaras1,

let me understand: do you want to filter events before indexing (index time) or during searches (search time)?

in the first case you lose all the other logs and you cannot use them for other searches (e.g. finding errors).

In this case you can follow the instructions at https://docs.splunk.com/Documentation/Splunk/9.0.3/Forwarding/Routeandfilterdatad#Filter_event_data_...

in few words, in props.conf:

[your_sourcetype]
TRANSFORMS-set= setnull,setparsing

in transforms.conf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX =  GET\s*\/\s*-\s*80|443
DEST_KEY = queue
FORMAT = indexQueue

if you want to filter events at search time, in other words only in displaying data, your search is almost correct: I'd use parenthesis:

index="*"  ("GET / - 80 -" OR "GET / - 443 -")

Ciao.

Giuseppe

[anandhalagaras1]

@gcusello ,

Thanks for your response.

Actually my query was not to ingest the logs if GET / - 80 or GET / - 443 is present in the IIS logs.

i.e. We want to filter (ignore) the events before indexing (index time) .

And we are already ingesting the IIS logs into Splunk from the servers but if the below event is present in the logs for the sourcetype "xyz" then it needs to be filtered out i.e. it is not required to be ingested into Splunk itself.

ie GET / - 80  or GET / - 443 is present in the IIS logs then it is not required to be ingested into Splunk. 

2022-12-12 00:38:10 xx.yyy.zzz.aaa GET / - 80 - xx.yyy.z.a - - x00 0 0 0

2022-12-12 00:44:45 xx.yyy.zzz.aaa GET / - 443 - xx.yyy.z.a - - y00 0 0 x

Hence kindly help with the props and transforms for the same.

[gcusello]

Hi @anandhalagaras1,

as you can see in my previous answer, you have to put the props.conf and transfroms.conf in the Indexers or (if present) on intermediate Heavy Forwarders, not on Universal Forwarders.

in the props.conf you have to indicate your sourcetype and add the row of TRANSFORMATION and in the transforms.conf you have to insert the rows I shared using the correct regex.

The one I shared should be correct but, test it.

ciao.

Giuseppe

[anandhalagaras1]

@gcusello , I used the below props and transforms in my Heavy Forwarder server to eliminate the logs if it contains "GET / - 80 -"  or "GET / - 443 -"  as you have provided but still i can see the events with GET / - 80 - and GET / - 443 - are still getting ingested into Splunk. 

So anything needs to be changed in props and transforms. Kindly let me know.

props.conf:
[sourcetype]
TRANSFORMS-set= setnull,setparsing

transforms.conf:
[setnull]
REGEX = GET\s*\/\s*-\s*80|443
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = .
DEST_KEY = queue
FORMAT = indexQueue

Also I have tried this props and transforms as well but still the same.

[sourcetype]
TRANSFORMS-null= setnull152,setnull153

[setnull152]
REGEX = GET\s*\/\s*-\s*80
DEST_KEY = queue
FORMAT = nullQueue

[setnull153]
REGEX = GET\s*\/\s*-\s*443
DEST_KEY = queue
FORMAT = nullQueue

But still its not working. Can you kindly help me.

[gcusello]

Hi @anandhalagaras1,

at first check the regexes in Splunk Search using the regex command.

Then One question: before the Heavy Forwarder where are located the conf files, are there other Heavy Forwarders?

I suppose that in the props.conf stanza header, you inserte the correct sourcetype value of the data source to filter.

Ciao.

Giuseppe

[anandhalagaras1]

@gcusello 

at first check the regexes in Splunk Search using the regex command. --> Yes I have checked in the Search Head and it seems to be fine.

index="*" sourcetype=**
| regex GET\s*\/\s*-\s*80

Then One question: before the Heavy Forwarder where are located the conf files, are there other Heavy Forwarders? --> The client machines forwarder output has been send to Heavy Forwarder 1 and from there it will reach the indexers so similarly I have placed the config files  props and transforms in an app  to the same Heavy Forwarder 1 server and also have restarted the services of the client machine as well but still the logs which contains 80 and 443 is not getting removed before ingestion.

I suppose that in the props.conf stanza header, you inserted the correct sourcetype value of the data source to filter.--> Yes exactly.

Currently I am using this props and transforms so is this a correct one to filter out the logs?

props.conf
[sourcetype]
TRANSFORMS-null= setnull152,setnull153

transforms.conf
[setnull152]
REGEX = GET\s*\/\s*-\s*80
DEST_KEY = queue
FORMAT = nullQueue

[setnull153]
REGEX = GET\s*\/\s*-\s*443
DEST_KEY = queue
FORMAT = nullQueue

[anandhalagaras1]

@gcusello ,

Thank you.

This solution worked for sometime. But now the issue is the logs are coming with "get" (In lower case) and the filtration applied for "GET" (Upper case) so the condition is that if the logs are coming with 80 or 443 with "get" OR "GET" the logs should not be ingested into Splunk. So kindly provide the required regex. 

Sample Logs:

2023-01-29 02:38:37 xx.yyy.zzz.aaa get / - 80 - qq.www.e.rrr - - abc d e f

2023-01-29 02:38:37 xx.yyy.zzz.aaa GET/ - 80 - qq.www.e.rrr - - abc d e f

Currently used props and transforms for reference.

props:

[sourcetype]
TRANSFORMS-null= setnull1002
TZ = Etc/GMT

transforms:

[setnull1002]
REGEX = GET\s*\/\s*-\s*80|443
DEST_KEY = queue
FORMAT = nullQueue

[gcusello]

Hi @anandhalagaras1,

you have to adapt the regex to this new requirement:

REGEX =  (GET|get)\s*\/\s*-\s*80|443

if one answer solves your need, please accept one answer for the other people of Community or tell me how I can help you.

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated 😉

[anandhalagaras1]

@gcusello , When i use this query to check in Splunk search head i am getting an error as "

Unknown search command 'get'.

index="xxx" | regex (GET|get)\s*\/\s*-\s*80|443

So kindly help to check and update.

[gcusello]

Hi @anandhalagaras1,

this regex is for the transforms.conf file,

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX =  (GET|get)\s*\/\s*-\s*80|443
DEST_KEY = queue
FORMAT = indexQueue

if you use it in a search you have to use quotes:

index="xxx" 
| regex "(GET|get)\s*\/\s*-\s*80|443"

 Ciao.

Giuseppe

[gcusello · Accepted solution]

Hi @anandhalagaras1,

maybe i misunderstood your requirement: do you want to take or discard events with 80 and 443?

if you want to take them and discard the others you can use my first solution:

in props.conf

[your_sourcetype]
TRANSFORMS-set= setnull,setparsing

in transormsconf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX =  GET\s*\/\s*-\s*80|443
DEST_KEY = queue
FORMAT = indexQueue

If you want to discard events with 80 and 443 you can use your solution.

Anyway, I don't like to use two stanzas, please try using only one, also because the sequence of commands (in props is relevant, so to avoid problems I prefer to have a single stanza.

Ciao.

Giuseppe