How to identify /var/log/secure from the syslog feed using REX?

[arangineni]

We are getting /var/log files monitored from the endpoints and sent to Splunk using syslog-ng on a single TCP port. Now when we see the vents in Splunk its combination of all the /var/log/secure, /var/log/messages etc. into a single index and syslog source type.

How can I identify just the /var/log/secure of out it and send to a other index from Splunk side? I am trying make use of props like below but not sure what the regex would be for identifying just /var/log/secure. Any help is appreciated.

####props

[linux_secure]

SOURCE_KEY = _raw

REGEX = ***Need REX for /var/log/secure events****

DEST_KEY = _MetaData:Index

FORMAT = abc_index

[PickleRick]

You can't match entries _from a given sourcefile_ just by regex on its contents. Especially that /var/log/secure can contain many different kinds of events.

You could try matching on specific program names but I wouldn't be so sure you can do a 1-1 mapping and that those programs don't also write to other logs.

Unfortunately, since you're pushing all events into a single logical stream you need - as you noticed yourself - another way of (de)multiplexing them. You could add the source filename or some identifier serving the same purpose on the sending syslog and then you'd have to "unpack" it and strip it from the event on the receiving end. I do this in my multi-layered rsyslog-based solution where I pass additional metadata between syslog servers on top of the event data itself.

But it's not as simple as "match this regex".

EDIT: OK, you could try to match your syslog rules sending various facility.priority combinations to different files to match the <PRI> header in your syslog events.

[woodcock]

Assuming that you are using a Splunk UF, deploy this configuration to it:

props.conf

[source::/var/log/secure]
index=YourIndexNameHere

[somesoni2]

Both data is getting logged in different files (source), so you can override index name using following source (index time configuration, must be set on instance where parsing happens)

props.conf

[source::/var/log/secure]
TRANSFORMS-index-linux_secure = index_linux_secure

transforms.conf

 [index_linux_secure]
 SOURCE_KEY = _raw 
 REGEX = . 
 DEST_KEY = _MetaData:Index 
 FORMAT = abc_index 

[arangineni]

Actually we DONT have source separation as well.

All the data is through syslog and source name is defined to a static name in our input. Basically source as syslog_feed and source type as syslog, that is where the challenging piece is. I need to now know the pattern for the secure logs and make use of the REGEX

[woodcock]

You are not giving us enough information to give you a good answer.  I agree with @somesoni2 so far.  The /var/log/secure stuff is a file in a path and surely you are picking it up using a Splunk UF on that box, right?  If so, EVEN IF you are doing an override of the source field, at the point that the UF is picking up the file, the original source value will exist and if you deploy his configurations as laid out to the UF, it should work.

[arangineni]

@woodcock @somesoni2 Here is more context:

Actually our setup flow is little different LIKE below. We are NOT using UF in out setup.

Endpoints(Includes all network appliances and Linux servers etc..)  ---> Central syslog-ng ---> Sending to Splunk HF on a TCP Port(At this point we have TCP input setup with sourcetype as syslog and source as syslog_feed).

Unfortunately we don't have option to install Splunk UF on syslog-ng and only way we can receive data is through TCP port onto Splunk HF

My requirement now is identify /var/log/secure events out of the whole feed send to a different index based on the event format using REGEX. Is there a specific Regex that can identify all the secure logs that I can use in my transforms below.

[linux_secure]

SOURCE_KEY = _raw

REGEX = ***Need REX to identify /var/log/secure events****

DEST_KEY = _MetaData:Index

FORMAT = abc_index

[woodcock]

In case I did not make it clear, @somesoni2 was correct about everything EXCEPT where to deploy the configuration change.  It must be on the UF, not the Indexers.

[PickleRick]

@woodcock @somesoni2 You're too fixed on this UF part 😉

@arangineniis _not_ using UF on the source machine. He's using some local syslog daemon that's sending the events from various local files to remote destination (in this case - to Splunk). I'd even risk a guess that he's not reading the files' contents, because that would be counterproductive, but rather sending the events as an additional action to writing to the local file.