Can Splunk DB Connect reformat data before indexing?

[ankithreddy777]

After we retrieve the data from the database, can Splunk DB Connect reformat data before indexing?

[aaraneta_splunk]

@ankithreddy777 - Were you able to test out nabeel652's solution? Did it work? If yes, please don't forget to resolve this post by clicking on "Accept". If you still need more help, please provide a comment with some feedback. Thanks!

[nabeel652]

You can do all sorts of reformatting through the SQL provided in the Inputs in DB Connect. So change all the formats like timestamp, field values based on case statement etc while providing the query in database inputs.

Below is my query that I use for getting SCOM ACS logs form the SCOM ACS DB through DB Connect. Note all sorts of data formatting done through convert and case commands:

SELECT 
 convert(varchar,dvA.CreationTime,120) + ' +0000' AS CreationTimeUTC,
 dvA.EventId,
 dvA.EventMachine AS Server,
 dvA.TargetUser AS Username,
CASE dvA.ClientDomain
   WHEN 'n/a' THEN PrimaryDomain
   ELSE dvA.ClientDomain
 END AS Domain,
 CASE dvA.String02
   WHEN 'n/a' THEN NULL
   ELSE dvA.String02
 END AS SourceMachine,
 dvA.CollectionTime
FROM
 AdtServer.dvAll5 AS dvA
WHERE
 dvA.EventId IN (4624,4648,4672,4634,4737,4740,4625,4767,4771,4738,644) AND CreationTime > convert(datetime, '2016-10-03 16:00:37.207')

[langlv]

I have same situation. There are fields in database which contain sensitive data and I dont want to show the clear text in Splunk. Instead, I am going to transform/encrypt those fields data before sending to Splunk for indexing.

Currently I am using DB Connect 2.x, and manually modify the code of the app (Python) to encrypt data before writing event to indexing stream. But since DB Connect 3.x I'm not able to do that.

Is there any way that I can add the custom handler for data stream from database, process it before return to DBX to submit to Splunk HEC ?

Thanks,

Lang

[langlv]

Be noted that I don't want to use any custom modification in database due to performance impact. I tried to apply encryption on fields in select query and it turned out very high CPU in database.

By moving data encryption to Splunk DBX, I can scale out the work load to a cluster of heavy forwarders.