How to exclude logs ingesting during index time

[bsuresh1]

In our IIS logs, we are getting thousands of lines like below which is of no use in ingesting into Splunk. So want to exclude these and want to ingest only the lines which has strings with data after GET/POST /xxxx.

Exclude to ingest:
2017-05-23 09:03:05 10.123.123.123 GET / - 80 - 10.123.123.123 - - 200 0 0 0
2017-05-23 09:02:28 10.123.123.123 POST / - 80 - 10.123.123.123 - - 200 0 0 0

Include to ingest:
2017-05-23 09:00:41 10.123.123.123 GET /applicationservice/V1/applicationservice wsdl=wsdl0 80 - 10.123.123.123 Java/1.7.0_25 - 200 0 0 0
2017-05-23 09:00:41 10.123.123.123 POST /applicationservice/V1/applicationservice - 80 - 10.123.123.123 JAX-WS+RI+2.2.4-b01 - 200 0 0 171

Please let me how to create props and transforms for this? I have tried below but it didn't work.

props:
[sourcetype::test_iis]
TRANSFORMS-set = setnull

transforms:
[setnull]
REGEX = \d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2} \d{1,3}.\d{1,3}.\d{1,3}.\d{1,3} GET [/] [-] 80 [-] \d{1,3}.\d{1,3}.\d{1,3}.\d{1,3} [-] [-] \d{1,3} \d{1,3} \d{1,3} \d{1,3}
DEST_KEY = queue
FORMAT = nullQueue

Below is my inputs.conf config:
[monitor://C:\temp\u*.log]
sourcetype = test_iis
index = testlog
disabled = 0

[harsmarvania57 · Accepted solution]

Try this in transforms.conf

[setnull]
REGEX = \d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2} \d{1,3}.\d{1,3}.\d{1,3}.\d{1,3} (GET|POST) [\/] [-].*
DEST_KEY = queue
FORMAT = nullQueue

[bsuresh1]

Thanks @harsmarvania57. Now the logs are being filtered after using this REGEX.

I have made another change to props.conf as below. Earlier it was like [sourcetype::test_iis]

[test_iis]
TRANSFORMS-set = setnull

[aaraneta_splunk]

@bsuresh1 - Please don't forget to "Accept" the answer by harsmarvania57 to close out your question. Thanks.

[DalJeanis]

The single mistake in your regex, according to regex101.com, is that the forward slash is not escaped.

Change [/] to [\/] and retry.

You can also do GET and POST in the same regex, like this

\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2} \d{1,3}.\d{1,3}.\d{1,3}.\d{1,3} (GET|POST) [\/] [-] 80 [-] \d{1,3}.\d{1,3}.\d{1,3}.\d{1,3} [-] [-] \d{1,3} \d{1,3} \d{1,3} \d{1,3}

[richgalloway]

Your REGEX string is more specific than it needs to be and doesn't match POST requests. Try this one: \d{1,3}.\d{1,3}.\d{1,3}.\d{1,3} (GET|POST) \/ - 80 -.

[bsuresh1]

I tried this in transforms but still it is ingesting all logs:
By the way, I have these files in an app(...\deployment-apps\iss\local) and pushing it from deployment manager. Is it the right place to do?

transforms:
[setnull]
REGEX = \d{1,3}.\d{1,3}.\d{1,3}.\d{1,3} (GET|POST) \/ - 80 -
DEST_KEY = queue
FORMAT = nullQueue

[setnull]
REGEX = (GET|POST)\s\/\s
DEST_KEY = queue
FORMAT = nullQueue

[gcusello]

Hi
you should try with you props.conf and the following transforms.conf

[setnull]
REGEX = (GET|POST)\s\/\s
DEST_KEY = queue
FORMAT = nullQueue

Bye.
Giuseppe

[bsuresh1]

I have tried the REGEX which was given cusello, but it is not working. Also tried the regex given by richgalloway, still it is ingesting all the data. I have kept all these files in /../deployment-apps/iis/local/. Is this the right place?

The changes I do in Deployment master is getting effect in the forwarder but still ingesting all logs.

[richgalloway]

So you're changing props.conf and transforms.conf on the forwarders? Try making the changes on your indexers.

[bsuresh1]

I have created an app (with props and transforms) and placed in Cluster master. Applied cluster bundle command but still all the logs are being ingested. Below is my latest props and transforms content:

props.conf:
[sourcetype::test_iis]
TRANSFORMS-set = setnull

transforms.conf:
[setnull]
REGEX = \d{1,3}.\d{1,3}.\d{1,3}.\d{1,3} (GET|POST) \/ - 80 -
DEST_KEY = queue
FORMAT = nullQueue