Transform.conf is not working

[agralrst]

I'm trying to extract field for Symantec ProxySG with transform.conf & props.conf but it isn't working.

Here is the sample logs:

Aug  4 16:31:58 2024-08-04 08: 31:28 "hostname" 5243 xx.xx.xx.xx 200 TCP_TUNNELED 6392 2962 CONNECT tcp domain.com 443 / - yyyy - xx.xx.xx.xx xx.xx.xx.xx "None" - - - - OBSERVED - - xx.xx.xx.xx - 7b711515341865e8-0000000008da5077-0000000066af3c5e - -

Here is my configuration: 

REGEX = ^.*"CN-SH-PSG-01"\s+(?<bytes_in>\d+)\s+(?<client_ip>\d+\.\d+\.\d+\.\d+)\s+(?<status_code>\d+)\s+(?<action>[^\s]+)\s+(?<bytes_out>\d+)\s+(?<bytes_out2>[^\s]+)\s+(?<http_method>[^\s]+)\s+(?<protocol>[^\s]+)\s+(?<domain>[^\s]+)\s+(?<port>\d+)\s+[^\s]+\s+(?<user>[^\s]+)\s+[^\s]+\s+[^\s]+\s+(?<mime_type>[^\s]+)\s+[^\s]+\s+"(?<user_agent>[^"]+)"FORMAT = bytes_in::$1 client_ip::$2 status_code::$3 action::$4 bytes_out::$5 bytes_out2::$6 http_method::$7 protocol::$8 domain::$9 port::$10 user::$11 mime_type::$12 user_agent::$13

 

 

[source::syslog]

TRANSFORMS-proxysg_field_extraction = proxysg_field_extraction

I've tried to change the config but the result teh field is not extracted & I have tried my regex using regex101.com and is doing fine

[richgalloway]

Put the props and transforms on the indexers and search heads.  If the forwarders are heavy forwarders then they go there, too.

What is the "Master Deployment"?  That's not a Splunk term.

[agralrst]

I mean it's deployment server & Master Node. 

Can I add the props.conf & transform.conf in deployment server and then run this command?

/opt/splunk/bin/splunk reload deploy-server

I'm sorry for asking the basic thing I'm new in Splunk.

[richgalloway]

Yes, you can run the reload deploy-server command and should install the props and transforms on the forwarders (if done correctly).  It will not install them on indexers or search heads.

[agralrst]

and the input is only configured in heavy forwarder B but when I put the transform.conf in HF B, it isn't working

[richgalloway]

There are many reasons why a transform might not work.  The most common is the regular expression doesn't match the incoming data.  It's also possible the source name doesn't match what's in props.conf so the transform is never invoked.

[agralrst]

anyway I have distributed deployment

- Master Depoloyment 

- Deployer

- INdexer A 

- Indexer B 

- License Master + Monitoring console

- Forwarder A 

- FOrwarder B 

- Search Head A 

- Search Head B

- Search Head C 

where should I add the props.conf and transform.conf?

[richgalloway]

The regex does not match the sample event.  It will only work for events from the host "CN-SH-PSG-01".  To match any host name, try this regex:

"(?:[\w-]+)"\s+(?<bytes_in>\d+)\s+(?<client_ip>\d+\.\d+\.\d+\.\d+)\s+(?<status_code>\d+)\s+(?<action>[^\s]+)\s+(?<bytes_out>\d+)\s+(?<bytes_out2>[^\s]+)\s+(?<http_method>[^\s]+)\s+(?<protocol>[^\s]+)\s+(?<domain>[^\s]+)\s+(?<port>\d+)\s+[^\s]+\s+(?<user>[^\s]+)\s+[^\s]+\s+[^\s]+\s+(?<mime_type>[^\s]+)\s+[^\s]+\s+"(?<user_agent>[^"]+)"

Notice I removed the meaningless "^.*" from the beginning.  That is implied in all regular expressions without the ^ anchor.

The FORMAT setting must be on a separate line, but I presume that's a copy-paste error.

Changes to transforms require a restart of the indexer and apply only to new events.

Make sure the source name associated with the data is exactly "syslog".   If it doesn't, try using the sourcetype name.