topic How to identify /var/log/secure from the syslog feed using REX? in Getting Data In

How to identify /var/log/secure from the syslog feed using REX?

arangineni — Wed, 23 Feb 2022 17:44:02 GMT

We are getting /var/log files monitored from the endpoints and sent to Splunk using syslog-ng on a single TCP port. Now when we see the vents in Splunk its combination of all the /var/log/secure, /var/log/messages etc. into a single index and syslog source type.

How can I identify just the /var/log/secure of out it and send to a other index from Splunk side? I am trying make use of props like below but not sure what the regex would be for identifying just /var/log/secure. Any help is appreciated.

####props

[linux_secure]

SOURCE_KEY = _raw

REGEX = ***Need REX for /var/log/secure events****

DEST_KEY = _MetaData:Index

FORMAT = abc_index

Re: Identifying /var/log/secure from the syslog feed using REX

somesoni2 — Tue, 22 Feb 2022 14:44:37 GMT

Both data is getting logged in different files (source), so you can override index name using following source (index time configuration, must be set on instance where parsing happens)

props.conf

[source::/var/log/secure]
TRANSFORMS-index-linux_secure = index_linux_secure

transforms.conf

 [index_linux_secure]
 SOURCE_KEY = _raw 
 REGEX = . 
 DEST_KEY = _MetaData:Index 
 FORMAT = abc_index

Re: Identifying /var/log/secure from the syslog feed using REX

arangineni — Tue, 22 Feb 2022 14:50:40 GMT

Actually we DONT have source separation as well.

All the data is through syslog and source name is defined to a static name in our input. Basically source as syslog_feed and source type as syslog, that is where the challenging piece is. I need to now know the pattern for the secure logs and make use of the REGEX

Re: Identifying /var/log/secure from the syslog feed using REX

woodcock — Wed, 23 Feb 2022 17:31:09 GMT

You are not giving us enough information to give you a good answer. I agree with @somesoni2 so far. The /var/log/secure stuff is a file in a path and surely you are picking it up using a Splunk UF on that box, right? If so, EVEN IF you are doing an override of the source field, at the point that the UF is picking up the file, the original source value will exist and if you deploy his configurations as laid out to the UF, it should work.

Re: Identifying /var/log/secure from the syslog feed using REX

woodcock — Wed, 23 Feb 2022 17:32:23 GMT

In case I did not make it clear, @somesoni2 was correct about everything EXCEPT where to deploy the configuration change. It must be on the UF, not the Indexers.

Re: Identifying /var/log/secure from the syslog feed using REX

woodcock — Wed, 23 Feb 2022 17:35:02 GMT

Assuming that you are using a Splunk UF, deploy this configuration to it:

props.conf

[source::/var/log/secure] index=YourIndexNameHere

Re: Identifying /var/log/secure from the syslog feed using REX

arangineni — Wed, 23 Feb 2022 17:50:20 GMT

@woodcock @somesoni2 Here is more context:

Actually our setup flow is little different LIKE below. We are NOT using UF in out setup.

Endpoints(Includes all network appliances and Linux servers etc..) ---> Central syslog-ng ---> Sending to Splunk HF on a TCP Port(At this point we have TCP input setup with sourcetype as syslog and source as syslog_feed).

Unfortunately we don't have option to install Splunk UF on syslog-ng and only way we can receive data is through TCP port onto Splunk HF

My requirement now is identify /var/log/secure events out of the whole feed send to a different index based on the event format using REGEX. Is there a specific Regex that can identify all the secure logs that I can use in my transforms below.

[linux_secure]

SOURCE_KEY = _raw

REGEX = ***Need REX to identify /var/log/secure events****

DEST_KEY = _MetaData:Index

FORMAT = abc_index

Re: How to identify /var/log/secure from the syslog feed using REX?

PickleRick — Wed, 23 Feb 2022 20:37:35 GMT

You can't match entries _from a given sourcefile_ just by regex on its contents. Especially that /var/log/secure can contain many different kinds of events.

You could try matching on specific program names but I wouldn't be so sure you can do a 1-1 mapping and that those programs don't also write to other logs.

Unfortunately, since you're pushing all events into a single logical stream you need - as you noticed yourself - another way of (de)multiplexing them. You could add the source filename or some identifier serving the same purpose on the sending syslog and then you'd have to "unpack" it and strip it from the event on the receiving end. I do this in my multi-layered rsyslog-based solution where I pass additional metadata between syslog servers on top of the event data itself.

But it's not as simple as "match this regex".

EDIT: OK, you could try to match your syslog rules sending various facility.priority combinations to different files to match the <PRI> header in your syslog events.

Re: Identifying /var/log/secure from the syslog feed using REX

PickleRick — Wed, 23 Feb 2022 20:35:40 GMT

@woodcock @somesoni2 You're too fixed on this UF part 😉

@arangineniis _not_ using UF on the source machine. He's using some local syslog daemon that's sending the events from various local files to remote destination (in this case - to Splunk). I'd even risk a guess that he's not reading the files' contents, because that would be counterproductive, but rather sending the events as an additional action to writing to the local file.