Can you look at our .conf files and see why our deployment app isn't working on our forwarder?

[wrangler2x]

We are using a deployment server to send a deployment app to a Splunk heavy forwarder. The forwarder is on a Windows Server 2002 R2 standard. The forwarder is on 6.5.2 Splunk. I've verified the Deployment App downloaded and it is correct in $SPLUNK_HOME\etc\apps\OIT_DA_syslogs_itsec_SecretServer_log

Here are the inputs, props, and transforms.conf files

inputs.conf

[monitor://D:\Web\log\SS.log]
disabled = false
index = syslogs_itsec
sourcetype = sslog

props.conf

[sslog]
TRANSFORMS-SSlogs = SSLogDrop, SSLogPass

transforms.conf

[SSLogDrop]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[SSLogPass]
REGEX = [\s'](?:MIMA01|mima01|ERROR\s+Thycotic\.AppCore\.Emailer2|RADIUS|Thycotic\.webapp\.Web\.usercontrols\.Login)[\s']
DEST_KEY = queue
FORMAT = indexQueue

If I go to regex101.com and past log entries that are in the ss.log file, and use this regex, all the expected things are getting highlighted. See https://regex101.com/r/ulfpsY/1

At one point, I just removed props.conf and transforms.conf and reloaded the Deployment App, and I later started seeing logs. Put them back in and then... nothing. I'm mystified.

[FrankVl]

Have you tried escaping the ' characters in your regex? so:

[SSLogPass]
  REGEX = [\s\'](?:MIMA01|mima01|ERROR\s+Thycotic\.AppCore\.Emailer2|RADIUS|Thycotic\.webapp\.Web\.usercontrols\.Login)[\s\']
  DEST_KEY = queue
  FORMAT = indexQueue

Don't think that should be necessary, but perhaps worth a try?

[harsmarvania57 · Accepted solution]

If Parameter name: context is in new line in actual log files then you can set SHOULD_LINEMERGE=false in props.conf for that particular sourcetype and it will work fine

2019-01-07 17:43:49,654   [107] ERROR Thycotic.AppCore.Emailer2 (null) - An error occured while sending an email (async internal): to: *redacted* subject: [SecretServer] Password Reset Exception: Value cannot be null.
Parameter name: context

If Parameter name: context is with same event then you can try below config in props.conf and transforms.conf will be same as provided by you. I have tested this in my lab environment and it is working fine.

props.conf

[sslog]
TRANSFORMS-SSlogs = SSLogDrop, SSLogPass
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\,\d{3}

[FrankVl]

And how exactly does that explain the issue that the SSLogPass transforms is not working?

[wrangler2x]

I added this to props.conf

SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\,\d{3}

and it started taking logs. I just got around to checking but I put this in last night before I left.

@FrankVl -- I don't think that an squote inside a character class in a regex should be a problem. It tested-out tokay on regex101.com. But just in case I went ahead and replaced the two of them with \x27 (which also works on regex101.com and in the splunk rex command). I also did that last night.

Anyway, thanks for the help and the logs are coming in just dandy!

[harsmarvania57]

Didn't get you and even I didn't modify SSLogPass transforms.

[FrankVl]

The question is about his filtering not working properly (all his events are dropped, if I understand the question correctly). How does changing the line breaker fix that? Maybe I'm missing something, but it appears to me that your answer is completely off-topic?

[harsmarvania57]

The answer which I provided is completely valid with the config and sample data @wrangler2x provided, even in my lab environment when I am using same config splunk is somehow merging all events and after that it is applying transforms and due to that not a single event is indexing.

Now if we look at sample data Parameter name: context is in new line, if I simply set SHOULD_LINEMERGE=false so splunk will not merge lines before applying transforms then it is extracting data based on REGEX provided in transforms but Parameter name: context is not matching with regex and as we used SHOULD_LINEMERGE=false splunk is treating that line as new event and it is discarding that event before indexing.

If Parameter name: context is not separate event so below lines should be a single event

2019-01-07 17:43:49,654   [107] ERROR Thycotic.AppCore.Emailer2 (null) - An error occured while sending an email (async internal): to: *redacted* subject: [SecretServer] Password Reset Exception: Value cannot be null.
Parameter name:  context

in this case we need to use SHOULD_LINEMERGE=false and LINE_BREAKER to break event at timestamp.

I hope this clears your query.

[FrankVl]

Ah, ok, now I get it 🙂

You didn't mention anywhere that this would actually fix the filtering because without line breaking settings it merges everything together. Never mind my comments then 🙂

[harsmarvania57]

No problem mate 🙂

[wrangler2x]

And this is the first time I've seen a case where a change in props.conf made a regex in transforms.conf work correctly. I'm definitely going to remember this one.

[harsmarvania57]

Hi @wrangler2x,

I am assuming that you are using Splunk Universal Forwarders on Windows server, in that case props.conf and transforms.conf for data parsing needs to be on Splunk Enterprise Instance (Either Indexer or Heavy Forwarder whichever comes first from UF) because most of the parsing happen on Indexer/Heavy Forwarder.

[wrangler2x]

This forwarder is a Heavy Forwarder. It should be able to parse these just fine.

[p_gurav]

Try using setnull stanza in transforms.conf:

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue 

[SSLogPass]
 REGEX = [\s'](?:MIMA01|mima01|ERROR\s+Thycotic\.AppCore\.Emailer2|RADIUS|Thycotic\.webapp\.Web\.usercontrols\.Login)[\s']
 DEST_KEY = queue
 FORMAT = indexQueue

and in props.conf:

 [sslog]
 TRANSFORMS-SSlogs = setnull, SSLogPass

[wrangler2x]

From what I've read, Splunk does not recommend using a generic transform stanza name like setnull because it could overwrite other transforms, but instead use a unique name (several comments about that in other Splunk Answers. But, just in case it there could be any difference, I made these changes and there was no effect, so I've set the drop one back to SSLogDrop.

I have many other Deployment Apps that do this same thing using various unique names and they all work fine. There must be something else going on here that needs fixing.