topic How do I emulate a tcp stream to test data ingestion? in Getting Data In

How do I emulate a tcp stream to test data ingestion?

jason0 — Mon, 03 Oct 2022 23:56:07 GMT

Hello,

I have a stream of call data records in xml form coming into splunk and i would like to add some ingestion-time transformations to it. However I have broken the input at least twice, so I need a debugging setup.

I ran a packet capture to get about three minutes worth of the stream (500 or so megabytes) and stripped out the xml data into a raw text file. I am going to "ingest" this file into a test server.

How do I dump the contents of an index so i can re-import the same data over and over again to test my transforms?

--jason

Re: How do I emulate a tcp stream to test data ingestion?

chaker — Tue, 04 Oct 2022 00:45:57 GMT

Hi @jason0

Take a look at this existing community answers. In summary, you need to clean the index, and reset the fishbucket pointer for the input you are testing.

Do this in a test environment. There is no undo for these steps.

https://community.splunk.com/t5/Getting-Data-In/btprobe-and-re-indexing-data/m-p/108265

https://community.splunk.com/t5/Deployment-Architecture/Use-btprobe-reset-to-re-index-multiple-files/td-p/313186

https://community.splunk.com/t5/Splunk-Search/Re-indexing-multiple-files-using-btprobe/td-p/298672

Hope this helps.