I've different log FTP files coming in. Each log file will be in a differnt format but with some common data across the log files. I wanted to extract those common information across the files and transform it into one source file AT INDEX TIME (not at search time).
I'm able to transform the data if it's from one source.
Please advise on how to combine the data from different files into one source file AT INDEX TIME.
UPDATE:
This is what we are trying to do AT THE INDEX TIME. The requirement is to use Splunk to only transform the data from different logs into a specified format(as mentioned below) only. We don't want to search any information. Additionally, please confirm in the below mentioned scenario if Splunk will be used as a Forwarder.
I have mentioned sample events from six different log files where each log file contains each information in it that . For example log 1 may contain the "IP address",log2 may contain the "No of bytes transferred" and such like that .I want to combine the logs together so that I can change the format of the logs to the below Ouput format
Log 1:
2013-11-22 00:03:06,124 [29682968] INFO: Secure storage keystore is disabled. [.utils.security.KeystoreReader]
2013-11-22 00:03:59,585 [68ea68ea] LOGON: There have been 1 logon failures for unrecognized userids in 0 seconds for these userids, IP addresses, and times: [null 167.113.182.41 11/22/13 0:3:59]. [.config.clientitf.FuncUser]
Log2:
2013-11-22 06:32:41,057 [29682968] FINE: Processing socket READ event [com.maverick.nio.SocketConnection]
2013-11-22 06:32:41,057 [29682968] FINE: 127845 bytes transferred
Log3:
13-11-22 05:26:54 [29682968] FINE: Received file c:/demo.csv
13-11-22 05:26:54 [5f7e5f7e] FINE: Processing socket READ event
Log4:
13-11-22 00:18:03 [69b269b2] INFO: Secure storage keystore is disabled.
13-11-22 00:33:03 [29682968] INFO: file c:/demo.csv in ascii mode.
Log5:
2013-11-22 00:03:19,224 [29682968] SUCCESS: Compressed the file demo.csv: systemid=27, machinename=rsbe.bnymellon.net, module=Transformation Service, description=Transformation Service [.alarms.heartbeatgenerator.GeneratorThread]
2013-11-22 00:10:19,523 [6d886d88] SUCCESS: Success, Alarm Heartbeat updated: systemid=27, machinename=rsbe.bnymellon.net, module=Transformation Service, description=Transformation Service [.alarms.heartbeatgenerator.GeneratorThread]
Log6:
2013-11-22 00:00:29,024 [51d451d4] SUCCESS: Scan successful: There are no documents for throttler Throttling Integration with owner ENCRYPT_DECRYPT to output. [.io.agents.throttle.ThrottleAgent]
2013-11-22 00:00:36,018 [516c516c] FINE: Scanning directory [/ftxprd1/BNYM_NONPROD_01/biz_outbound] [.io.agents.dirmon.DirMonInbound]
2013-11-22 00:00:36,018 [516c516c] FINE: Scanning dir [/ftxprd1/BNYM_NONPROD_01/biz_outbound] [.io.agents.dirmon.DirMonInbound]
Log7:
2013-11-22 00:03:07,734 [19581958] SUCCESS: Success, Alarm Heartbeat updated: systemid=25, machinename=rsbe.bnymellon.net, module=Inbound Listeners, description=Inbound Listeners [.alarms.heartbeatgenerator.GeneratorThread]
2013-11-22 00:10:08,023 [29682968] SUCCESS: Success, Alarm Heartbeat updated: systemid=25, machinename=rsbe.bnymellon.net, module=Inbound Listeners, description=Inbound Listeners [.alarms.heartbeatgenerator.GeneratorThread]
... View more