topic Need help in extraction of data from distributed and async logging application in Splunk Search

Need help in extraction of data from distributed and async logging application

Anurag101 — Mon, 22 Jan 2024 16:10:58 GMT

Hi All,

I am almost a starter in Splunk but my org uses this tool as a log management utility.

I need help in getting a direction so as to how to filter data from logs in a distributed a sync logging product.

Problem statement:

There are multiple log files on multiple Linux boxes getting generated every second.

I need to search for ids created and relevant creation timestamps and the batches under which these ids exists.
Filter the ids based on passed batches (this is another line in the same log file)
Calculate the E2E timestamp for the id processing by searching the processed id in step-1 and substracting the timestamp of step 3 and step 1(this is again printed in the log files).

I have been doing this using oracle external tables and Linux shell but need to do it in a better way using Splunk and need help, opinion's are highly appreciated

Re: Need help in extraction of data from distributed and async logging application

scelikok — Tue, 23 Jan 2024 17:44:53 GMT

Hi @Anurag101,

It is not possible to help without knowing your data. Also, the solution depends on whether data is extracted or not. If you can show some sample anonymized events, we can help by sample query.

Re: Need help in extraction of data from distributed and async logging application

PickleRick — Tue, 23 Jan 2024 19:09:06 GMT

As @scelikok said, your description is a bit vague and without a sample (anonymized if needed - we don't need your internal secrets ;-)) of your data and description of what data you want to get from that it's pretty much impossible to help you because we have no idea what we're talking about.