Configure Splunk to collect data around a spesific occurrence

[avitallange]

Hi,

Is it possible to configure Splunk so that if an error trace occurs, it will start collecting info traces around the error?
For example: error trace occurs at 1:00PM. at 1:00 PM Splunk will start to collect info traces since 12:45 PM till 1:15 PM.
Errors and info can be saved in distinct files.
I am asking because the info traces are too noisy and we would like to collect them according specific needs.

Thanks,
Avital

[avital]

The concern is about unnecessarily consume your licence

[mbenwell]

Is the concern about noise because it will unnecessarily consume your licence?

Or are you more interested in how to construct a search query to do the above?

[Gilberto_Castil]

Nope. This is not something that can be done on demand. The main reason that a monitor input is designed to read data from a file or files and keep telemetry on the read position in that file. This is what allows the monitor input to perpetually tail a file. This type of operation is done without any analysis of the data being read. It is just a blind consumption based on the read position on the file.

It is important to understand that for a monitor input to read a specific chunk of data, there must be a record of the seek pointer to where the data starts. To accomplish the rest of the task, it would also be required to know where in the file the data ends. However, this is not how the read functionality is designed. Each file is read from the beginning to the end and the data is entered into the input pipeline accordingly.

There are some absolutely mind-altering reads on the fishbucket in the blogs. If you need to dig more on the process, you may want to do a bit more digging starting here.

---

The only advice that may be applicable here is to have your application create rotational logs for shorter periods of time… say one hour. The train of thought is to have an alert which will trigger the activation of a monitor input which will capture only the active log. The activation can be done using the Deployment Server and a Splunk CLI command through a scripted alert, remote SSH, or whatever technical telepathy that suits your skills.

You would also need to know when to shut that special monitor off.

This path, however, will require you to test the procedure before you go and change the way your application creates its logs.

[billford]

I think you could do this with an alert, script and the sdk. I don't think there's any native way to do it but I'd be thrilled to be wrong.