Getting Data In

Possible to add a complete file (oneshot/sinkhole) AND followTail

nocostk
Communicator

I'd like to start monitoring a file that has been around for a while. I need to get all the older data in the file AND start tailing it (or inversely start tailing it and then realize I need the older data, too). What is the easiest way to do this?

Here is the inputs.conf

[monitor:///opt/scripts/sendShipConf/sendShipConf_cron.log]
disabled = false
followTail = 1
sourcetype = sendshipconf

I tried to oneshot it but the older info isn't there:

/opt/splunk/bin/splunk add oneshot /opt/scripts/sendShipConf/sendShipConf_cron.log

I could use a sinkhole but the sinkhole is defined globally (which means that I can't [or rather, don't want to] apply specific sourcetypes to it). This is from my "common" (global) app pushed to each forwarder:

[batch:///opt/splunk/var/spool/splunk]
move_policy = sinkhole
Tags (2)
0 Karma

mayler
Path Finder

I'd agree with Mick. Wouldn't nocostk just need to add a new data input, file, source: monitor file or directory, /path/to/file, make sure "follow tail" is de-selected.

0 Karma

mayler
Path Finder

if you set it to 1, wouldn't it only index the new data? setting it to 0 would index the old data and new data. no?

0 Karma

nocostk
Communicator

Right. I could just set followTail = 0, but for the sake of consistency across my environments I always just enable that value. I'd prefer to be able to add retroactive events on the fly.

0 Karma

Mick
Splunk Employee
Splunk Employee

I think you may be making this a bit over-complicated, it sounds like you just need to add a regular monitor input, that will capture all of the old data and any new incoming data going forward.

Or is there some detail that I'm missing here?

nocostk
Communicator

Well, maybe so. Here's a scenario: A developer asks me to start indexing a log. I get that setup and Splunk is happily chugging away at it (followTail). Developer emails me later and asks "is this all the older data or just the new data? How can I get the older data, too?".

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...