Getting Data In

Monitoring a file that is continiouisly written to

horsefez
SplunkTrust
SplunkTrust

Hi fellow splunkers,

I got the task to monitor a file on a system that gets created on serverstart and then gets written to for the time the server runs (6 months or more)

How am I able to monitor such a large file if new lines get added?
I heard about "follow tail", but anyone seems to discourage you to do that.

Any idea how this could be possibly done?

Thanks in advance!
Best regards,
pyro_wood

0 Karma
1 Solution

skoelpin
SplunkTrust
SplunkTrust

If I'm understanding the question correctly, you want to monitor a file continuously and send that data to Splunk? If so then you can install a universal forwarder to monitor that file. Each time that file has new data, the Splunk forwarder will see this and forward data to the indexer.

So an example would be, you have a file that sits on a file system and gets written to when it starts and if there is data requested from the server. Once this file is written to, the Splunk forwarder will see this and pick up all new changes and forward it to Splunk while disregarding everything it's already indexed from that file. You can write to the file as little or frequently as you want to

If you want to monitor a file, your inputs.conf will be located in \etc\system\local and look like this

[monitor//C:\PATH_TO_FILE]
disabled=false
sourcetype=YOUR_SOURCETYPE
index=YOUR_INDEX

And your output.conf will point to the indexer and will look like this

[tcpout]
defaultGroup = INDEXER_IP_9997

[tcpout:INDEXER_IP_9997]
server = INDEXER_IP:9997

[tcpout-server://INDEXER_IP:9997]

View solution in original post

0 Karma

woodcock
Esteemed Legend

What is wrong with a simple monitor stanza? This is exactly what Splunk is designed to do. How is your situation in any way complicated/non-standard?

0 Karma

horsefez
SplunkTrust
SplunkTrust

Looks like you are correct. Had a faulty monitor-stanza!
Thanks!

0 Karma

skoelpin
SplunkTrust
SplunkTrust

If I'm understanding the question correctly, you want to monitor a file continuously and send that data to Splunk? If so then you can install a universal forwarder to monitor that file. Each time that file has new data, the Splunk forwarder will see this and forward data to the indexer.

So an example would be, you have a file that sits on a file system and gets written to when it starts and if there is data requested from the server. Once this file is written to, the Splunk forwarder will see this and pick up all new changes and forward it to Splunk while disregarding everything it's already indexed from that file. You can write to the file as little or frequently as you want to

If you want to monitor a file, your inputs.conf will be located in \etc\system\local and look like this

[monitor//C:\PATH_TO_FILE]
disabled=false
sourcetype=YOUR_SOURCETYPE
index=YOUR_INDEX

And your output.conf will point to the indexer and will look like this

[tcpout]
defaultGroup = INDEXER_IP_9997

[tcpout:INDEXER_IP_9997]
server = INDEXER_IP:9997

[tcpout-server://INDEXER_IP:9997]
0 Karma

horsefez
SplunkTrust
SplunkTrust

Thank you skoelpin,
I did make an error in the monitor-stanza so the file didn't get read correctly. So I wondered if splunk wasn't able to do this task. Thanks to your example I was able to realize and fix this!

gcusello
SplunkTrust
SplunkTrust

What is your problem? Can you explain it?
Splunk usually manage upgrade of a single file with insert of new lines, without particular configurations.
Are you using a Forwarder or is a local file?

Bye.
Giuseppe

0 Karma

horsefez
SplunkTrust
SplunkTrust

Hey cusello,
you are right. It was my fault, had a faulty monitor-stanza!

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...