Getting Data In

How to index certain logs only during a certain time range (6am - 6pm)?

agoktas
Communicator

Hello,

I have 4 log files on one Host that I want to index/ingest.

Log #1, #2, #3 will be ingested 24 hours a day, but log file #4 shares a batch process in the evening that has 20 - 30GB per evening of events that is not needed, nor do we want to pay for - because I wouldn't use them at this point in time.

I want to avoid stopping the Splunk Universal Forwarder Windows service from 6pm to 6am because that would mean that logs #1, #2, & #3 will not index. Also, I believe it would pool up in the fish bucket anyway, so that will null my effort to exclude indexing from 6pm to 6am for log #4.

Any ideas how I can avoid indexing log #4 from 6pm to 6am (night time batch window)?

Thanks!

Tags (2)
0 Karma
1 Solution

MuS
SplunkTrust
SplunkTrust

Hi agoktas,

there are multiple ways of achieve this:

Update:
Following up all comments, this was the final working config:

Here is the final config that looks to be working great (we forgot '00' for the midnight hour):
Indexer configuration:

props.conf

[AppInternal]
TRANSFORMS-null= Appsetnull

transforms.conf

#Discard all events between 6pm - 6am
[Appsetnull]
REGEX = (?:d+/d+/d+|d+-d+-d+)s(18|19|20|21|22|23|00|01|02|03|04|05):
DEST_KEY = queue
FORMAT = nullQueue

Hope this helps ...

cheers, MuS

View solution in original post

Richfez
SplunkTrust
SplunkTrust

To add to MuS's answer, which probably has the right technique;

If you can't identify events by some string to use nullQueue consistently and easily, ... Wait, MuS, you'll probably know this better than me, but can you possibly do it on a match of time strings? As per your link to the docs, yiou could just set your REGEX to something that will match hours 18-23 and 0-5. (Possibly more easily done in two separate stanzas). You'll have to be careful, and you'll have to have very consistently identifiable timestamps, but I think this might be the way to do it if this is possible.

Beyond that, here are some other options and comments that I came up:

Any technique that stops/starts/changes the local forwarder seems like it won't work unless you do at least some local file manipulation. As MuS says, something like "erasing" the log4 file could do it if that were possible in a reasonable timeframe, so your process would be to stop the UF while the process runs for an hour, clear out the file then restart the UF. That will interrupt the other file inputs for the duration, but they'll catch up.

Along that same thread but with less interruption to the other files, you could stop the forwarder, change the config to ignore file 4 and restart. At the end of the period, you could then "erase" file 4, fix the config back and restart the forwarder. You still have to erase or remove file 4 or else the forwarder will just grab it all anyway. You could just rename it so it no longer matches the inputs before your last restart.

For the nullQueue side, same sort of technique as the first may work. At the beginning of your nullQueue period you could stop the forwarder, copy a config into place that does the nullQueue on file4's inputs, then start it back up. At the end, reverse the process. That way you would have a nearly uninterrupted ingest of files1-3 (and even file4), but the period you don't want would have file 4's contents being sent to the big bit bucket in the sky. This does mean you have another Splunk instance of some sort restarting twice a day, possibly interrupting something else.

Let us know if any of these help, or if one sounds particularly wonderful but you need a little help implementing it!

MuS
SplunkTrust
SplunkTrust

great idea to filter on the time! Thanks for this hint!

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Laser Bananas and Edge Hubs: Exploring Operational Technology (OT) Data Through a ...

  OT is a different environment to traditional IT and can have interesting challenges when interfacing the ...

Event Series: Mastering AI Tokenomics and Splunk Agent Observability

Beyond the Black Box: Correlating AI Performance and Tokenomics with Splunk Agent Observability   As ...

span_metrics: The OpenTelemetry-Idiomatic Way to See Inside Your Services

You open a trace in Splunk Observability Cloud and everything looks fine. One root span, order-pipeline, with ...