Hi, We have implemented a custom command which queries the external rest api and pulls the data to splunk search page. The challenge we are facing is when the response data is huge Splunk search page is waiting for couple of minutes(more than 5 minutes) with out showing any data. The result of the api's comes in the form of partitions. Lets say, if we have 100k records in the api rsult, All those 100k rows would be splitting up into 100 partitions and we need to iterate over 100 times to get all the 100k records. Similarly if we could send the partition data to splunk and get the results appended to the splunk page as and when we get data for all partitions, we can have end user see the data as soon as possible instead of waiting for couple of minutes. My custom command is generating custom command. I would like to know if there is any way to send the data in chunks to Splunk page instead waiting to pull all the 100k records. We tried couple of ways like yield (our code is in Python and using Splunk python SDK), enabling streaming attribute etc. Please help me here to figure out a way to send the data in chunks for the generating custom command. Thanking you. ... View more

Hi, I have an use case that I need to forward the logs to TCP listener (third party system) as and when the logs load to splunk. For the, I am loading windows security logs to splunk from windows machine using Splunk Forward service. Then configured splunk forwarder to forward raw logs to TCP listner as below. ================= props.conf =============== [ tcp:9080] TRUNCATE = 0 [default] # unless a more specific stanza clears the value of this class, the transform will be run TRANSFORMS-selectiveIndexing = selectiveIndexing [WinEventLog:Security] TRANSFORMS-routing=transforms-windows-security-logs # note the empty list of transforms to run in this class, overridden from the [default] TRANSFORMS-selectiveIndexing = =========================================   ================ transforms.conf =========== [transforms-windows-security-logs] REGEX = . DEST_KEY = _TCP_ROUTING FORMAT = windows-security-routing =========================================   ================ output.conf =============== [indexAndForward] index = true [tcpout] defaultGroup=everythingElseGroup [tcpout:windows-security-routing] server=xx.xx.xxx.xxx:9522 sendCookedData=false ======================================== Now, I have Logstash listener which is listening on port 9522 and writing the data to file. I see the logs being written to file as below =============== Windows security logs forwarded to third party system ========= {"@timestamp":"2022-05-24T10:15:57.680Z","@version":"1","port":33332,"message":"LogName=Security","host":"xx.xx.xxx.xxx"} {"@timestamp":"2022-05-24T10:15:57.680Z","@version":"1","port":33332,"message":"EventType=0","host":"xx.xx.xxx.xxx"} {"@timestamp":"2022-05-24T10:15:57.680Z","@version":"1","port":33332,"message":"SourceName=Microsoft Windows security auditing.","host":"xx.xx.xxx.xxx"} {"@timestamp":"2022-05-24T10:15:57.680Z","@version":"1","port":33332,"message":"RecordNumber=24054056","host":"xx.xx.xxx.xxx"} {"@timestamp":"2022-05-24T10:15:57.680Z","@version":"1","port":33332,"message":"TaskCategory=Process Termination","host":"xx.xx.xxx.xxx"} {"@timestamp":"2022-05-24T10:15:57.680Z","@version":"1","port":33332,"message":"Message=A process has exited.\r","host":"xx.xx.xxx.xxx"} {"@timestamp":"2022-05-24T10:15:57.680Z","@version":"1","port":33332,"message":"Subject:\r","host":"xx.xx.xxx.xxx"} {"@timestamp":"2022-05-24T10:15:57.680Z","@version":"1","port":33332,"message":"\tAccount Name:\t\tBDELYSYS07$\r","host":"xx.xx.xxx.xxx"} {"@timestamp":"2022-05-24T10:15:57.680Z","@version":"1","port":33332,"message":"\tLogon ID:\t\t0x3E7\r","host":"xx.xx.xxx.xxx"} {"@timestamp":"2022-05-24T10:15:57.680Z","@version":"1","port":33332,"message":"Process Information:\r","host":"xx.xx.xxx.xxx"} {"@timestamp":"2022-05-24T10:15:57.680Z","@version":"1","port":33332,"message":"\tProcess Name:\tC:\\Program Files\\SplunkUniversalForwarder\\bin\\splunk-winevtlog.exe\r","host":"xx.xx.xxx.xxx"} {"@timestamp":"2022-05-24T10:15:57.680Z","@version":"1","port":33332,"message":"05/23/2022 08:51:21 PM","host":"xx.xx.xxx.xxx"} {"@timestamp":"2022-05-24T10:15:57.681Z","@version":"1","port":33332,"message":"EventCode=4689","host":"xx.xx.xxx.xxx"} {"@timestamp":"2022-05-24T10:15:57.681Z","@version":"1","port":33332,"message":"ComputerName=BDELYSYS07.bdelysium.internal","host":"xx.xx.xxx.xxx"} {"@timestamp":"2022-05-24T10:15:57.681Z","@version":"1","port":33332,"message":"Type=Information","host":"xx.xx.xxx.xxx"} {"@timestamp":"2022-05-24T10:15:57.681Z","@version":"1","port":33332,"message":"Keywords=Audit Success","host":"xx.xx.xxx.xxx"} {"@timestamp":"2022-05-24T10:15:57.681Z","@version":"1","port":33332,"message":"OpCode=Info","host":"xx.xx.xxx.xxx"} {"@timestamp":"2022-05-24T10:15:57.681Z","@version":"1","port":33332,"message":"\r","host":"xx.xx.xxx.xxx"} {"@timestamp":"2022-05-24T10:15:57.681Z","@version":"1","port":33332,"message":"\tSecurity ID:\t\tS-1-5-18\r","host":"xx.xx.xxx.xxx"} {"@timestamp":"2022-05-24T10:15:57.681Z","@version":"1","port":33332,"message":"\tAccount Domain:\t\tBDELYSIUM\r","host":"xx.xx.xxx.xxx"} {"@timestamp":"2022-05-24T10:15:57.681Z","@version":"1","port":33332,"message":"\r","host":"xx.xx.xxx.xxx"} {"@timestamp":"2022-05-24T10:15:57.681Z","@version":"1","port":33332,"message":"\tProcess ID:\t0x2798\r","host":"xx.xx.xxx.xxx"} {"@timestamp":"2022-05-24T10:15:57.681Z","@version":"1","port":33332,"message":"\tExit Status:\t0x0","host":"xx.xx.xxx.xxx"} {"@timestamp":"2022-05-24T10:15:57.681Z","@version":"1","port":33332,"message":"LogName=Security","host":"xx.xx.xxx.xxx"} ================================================= Going through the logs, it seems like logs are not forwarded event by event.  I would like to know with the above setup, how can I forward the logs to TCP listener event by event. Also, is there a way to forward the splunk parsed data logs in JSON format to TCP port. ... View more