Getting Data In

How to exclude logs ingesting during index time

bsuresh1
Path Finder

In our IIS logs, we are getting thousands of lines like below which is of no use in ingesting into Splunk. So want to exclude these and want to ingest only the lines which has strings with data after GET/POST /xxxx.

Exclude to ingest:
2017-05-23 09:03:05 10.123.123.123 GET / - 80 - 10.123.123.123 - - 200 0 0 0
2017-05-23 09:02:28 10.123.123.123 POST / - 80 - 10.123.123.123 - - 200 0 0 0

Include to ingest:
2017-05-23 09:00:41 10.123.123.123 GET /applicationservice/V1/applicationservice wsdl=wsdl0 80 - 10.123.123.123 Java/1.7.0_25 - 200 0 0 0
2017-05-23 09:00:41 10.123.123.123 POST /applicationservice/V1/applicationservice - 80 - 10.123.123.123 JAX-WS+RI+2.2.4-b01 - 200 0 0 171

Please let me how to create props and transforms for this? I have tried below but it didn't work.

props:
[sourcetype::test_iis]
TRANSFORMS-set = setnull

transforms:
[setnull]
REGEX = \d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2} \d{1,3}.\d{1,3}.\d{1,3}.\d{1,3} GET [/] [-] 80 [-] \d{1,3}.\d{1,3}.\d{1,3}.\d{1,3} [-] [-] \d{1,3} \d{1,3} \d{1,3} \d{1,3}
DEST_KEY = queue
FORMAT = nullQueue

Below is my inputs.conf config:
[monitor://C:\temp\u*.log]
sourcetype = test_iis
index = testlog
disabled = 0

1 Solution

harsmarvania57
SplunkTrust
SplunkTrust

Try this in transforms.conf

[setnull]
REGEX = \d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2} \d{1,3}.\d{1,3}.\d{1,3}.\d{1,3} (GET|POST) [\/] [-].*
DEST_KEY = queue
FORMAT = nullQueue

View solution in original post

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Try this in transforms.conf

[setnull]
REGEX = \d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2} \d{1,3}.\d{1,3}.\d{1,3}.\d{1,3} (GET|POST) [\/] [-].*
DEST_KEY = queue
FORMAT = nullQueue

View solution in original post

0 Karma

bsuresh1
Path Finder

Thanks @harsmarvania57. Now the logs are being filtered after using this REGEX.

I have made another change to props.conf as below. Earlier it was like [sourcetype::test_iis]

[test_iis]
TRANSFORMS-set = setnull

0 Karma

aaraneta_splunk
Splunk Employee
Splunk Employee

@bsuresh1 - Please don't forget to "Accept" the answer by harsmarvania57 to close out your question. Thanks.

0 Karma

DalJeanis
SplunkTrust
SplunkTrust

The single mistake in your regex, according to regex101.com, is that the forward slash is not escaped.

Change [/] to [\/] and retry.

You can also do GET and POST in the same regex, like this

\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2} \d{1,3}.\d{1,3}.\d{1,3}.\d{1,3} (GET|POST) [\/] [-] 80 [-] \d{1,3}.\d{1,3}.\d{1,3}.\d{1,3} [-] [-] \d{1,3} \d{1,3} \d{1,3} \d{1,3}
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Your REGEX string is more specific than it needs to be and doesn't match POST requests. Try this one: \d{1,3}.\d{1,3}.\d{1,3}.\d{1,3} (GET|POST) \/ - 80 -.

---
If this reply helps you, an upvote would be appreciated.
0 Karma

bsuresh1
Path Finder

I tried this in transforms but still it is ingesting all logs:
By the way, I have these files in an app(...\deployment-apps\iss\local) and pushing it from deployment manager. Is it the right place to do?

transforms:
[setnull]
REGEX = \d{1,3}.\d{1,3}.\d{1,3}.\d{1,3} (GET|POST) \/ - 80 -
DEST_KEY = queue
FORMAT = nullQueue

[setnull]
REGEX = (GET|POST)\s\/\s
DEST_KEY = queue
FORMAT = nullQueue

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi
you should try with you props.conf and the following transforms.conf

[setnull]
REGEX = (GET|POST)\s\/\s
DEST_KEY = queue
FORMAT = nullQueue

Bye.
Giuseppe

0 Karma

bsuresh1
Path Finder

I have tried the REGEX which was given cusello, but it is not working. Also tried the regex given by richgalloway, still it is ingesting all the data. I have kept all these files in /../deployment-apps/iis/local/. Is this the right place?

The changes I do in Deployment master is getting effect in the forwarder but still ingesting all logs.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

So you're changing props.conf and transforms.conf on the forwarders? Try making the changes on your indexers.

---
If this reply helps you, an upvote would be appreciated.
0 Karma

bsuresh1
Path Finder

I have created an app (with props and transforms) and placed in Cluster master. Applied cluster bundle command but still all the logs are being ingested. Below is my latest props and transforms content:

props.conf:
[sourcetype::test_iis]
TRANSFORMS-set = setnull

transforms.conf:
[setnull]
REGEX = \d{1,3}.\d{1,3}.\d{1,3}.\d{1,3} (GET|POST) \/ - 80 -
DEST_KEY = queue
FORMAT = nullQueue

0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.