topic Re: Transform.conf is not working in Other Usage

Transform.conf is not working

agralrst — Sun, 04 Aug 2024 13:18:45 GMT

I'm trying to extract field for Symantec ProxySG with transform.conf & props.conf but it isn't working.

Here is the sample logs:

Aug 4 16:31:58 2024-08-04 08: 31:28 "hostname" 5243 xx.xx.xx.xx 200 TCP_TUNNELED 6392 2962 CONNECT tcp domain.com 443 / - yyyy - xx.xx.xx.xx xx.xx.xx.xx "None" - - - - OBSERVED - - xx.xx.xx.xx - 7b711515341865e8-0000000008da5077-0000000066af3c5e - -

Here is my configuration:

REGEX = ^.*"CN-SH-PSG-01"\s+(?<bytes_in>\d+)\s+(?<client_ip>\d+\.\d+\.\d+\.\d+)\s+(?<status_code>\d+)\s+(?<action>[^\s]+)\s+(?<bytes_out>\d+)\s+(?<bytes_out2>[^\s]+)\s+(?<http_method>[^\s]+)\s+(?<protocol>[^\s]+)\s+(?<domain>[^\s]+)\s+(?<port>\d+)\s+[^\s]+\s+(?<user>[^\s]+)\s+[^\s]+\s+[^\s]+\s+(?<mime_type>[^\s]+)\s+[^\s]+\s+"(?<user_agent>[^"]+)"FORMAT = bytes_in::$1 client_ip::$2 status_code::$3 action::$4 bytes_out::$5 bytes_out2::$6 http_method::$7 protocol::$8 domain::$9 port::$10 user::$11 mime_type::$12 user_agent::$13 [source::syslog] TRANSFORMS-proxysg_field_extraction = proxysg_field_extraction

I've tried to change the config but the result teh field is not extracted & I have tried my regex using regex101.com and is doing fine

Re: Transform.conf is not working

richgalloway — Sun, 04 Aug 2024 13:40:20 GMT

The regex does not match the sample event. It will only work for events from the host "CN-SH-PSG-01". To match any host name, try this regex:

"(?:[\w-]+)"\s+(?<bytes_in>\d+)\s+(?<client_ip>\d+\.\d+\.\d+\.\d+)\s+(?<status_code>\d+)\s+(?<action>[^\s]+)\s+(?<bytes_out>\d+)\s+(?<bytes_out2>[^\s]+)\s+(?<http_method>[^\s]+)\s+(?<protocol>[^\s]+)\s+(?<domain>[^\s]+)\s+(?<port>\d+)\s+[^\s]+\s+(?<user>[^\s]+)\s+[^\s]+\s+[^\s]+\s+(?<mime_type>[^\s]+)\s+[^\s]+\s+"(?<user_agent>[^"]+)"

Notice I removed the meaningless "^.*" from the beginning. That is implied in all regular expressions without the ^ anchor.

The FORMAT setting must be on a separate line, but I presume that's a copy-paste error.

Changes to transforms require a restart of the indexer and apply only to new events.

Make sure the source name associated with the data is exactly "syslog". If it doesn't, try using the sourcetype name.

Re: Transform.conf is not working

agralrst — Sun, 04 Aug 2024 15:33:49 GMT

anyway I have distributed deployment

- Master Depoloyment

- Deployer

- INdexer A

- Indexer B

- License Master + Monitoring console

- Forwarder A

- FOrwarder B

- Search Head A

- Search Head B

- Search Head C

where should I add the props.conf and transform.conf?

Re: Transform.conf is not working

richgalloway — Sun, 04 Aug 2024 16:52:46 GMT

Put the props and transforms on the indexers and search heads. If the forwarders are heavy forwarders then they go there, too.

What is the "Master Deployment"? That's not a Splunk term.

Re: Transform.conf is not working

agralrst — Mon, 05 Aug 2024 02:54:40 GMT

I mean it's deployment server & Master Node.

Can I add the props.conf & transform.conf in deployment server and then run this command?

/opt/splunk/bin/splunk reload deploy-server

I'm sorry for asking the basic thing I'm new in Splunk.

Re: Transform.conf is not working

agralrst — Mon, 05 Aug 2024 02:56:53 GMT

and the input is only configured in heavy forwarder B but when I put the transform.conf in HF B, it isn't working

Re: Transform.conf is not working

richgalloway — Mon, 05 Aug 2024 12:26:28 GMT

Yes, you can run the reload deploy-server command and should install the props and transforms on the forwarders (if done correctly). It will not install them on indexers or search heads.

Re: Transform.conf is not working

richgalloway — Mon, 05 Aug 2024 12:28:11 GMT

There are many reasons why a transform might not work. The most common is the regular expression doesn't match the incoming data. It's also possible the source name doesn't match what's in props.conf so the transform is never invoked.