Getting Data In

Configure input.txt Universal forwarder

sieutruc
Contributor

Hello,

I create an idexer server as server01 and a Universal forwarder that monitors and forwards a file in real time to indexer "example" of the indexer with the configuration below:
C:\Program Files\SplunkUniversalForwarder\etc\system\local\input.conf

[monitor://D:\AFCSystem\log\log_file.txt]
disabled = 0
_TCP_ROUTING = splunkserver01_9997
index = example
followTail = 1

and output.conf at the same directory

[tcpout]
defaultGroup = splunkserver01_9997

[tcpout:splunkserver01_9997]
disabled = false
server = splunkserver01:9997

[tcpout-server://splunkserver01:9997]

But when i view it in "example" indexer, i didn't see anything ?
Do you know the problem ?

Tags (1)
0 Karma
1 Solution

kristian_kolb
Ultra Champion

A couple of things,

If you call the file inputs.txt you will have problems, it should be called inputs.conf, otherwise Splunk will not recognize it. The same goes for outputs.conf

The order of precedence can be found in the docs.

You must configure the indexer to listen on port 9997. (manager -> forwarding and receiving -> configure receiving -> new).

Make sure that you have no firewalls in between (or local), blocking the traffic.

By the way, I don't think you'll need the _TCP_ROUTING stuff in the form you've written it;

(from the docs on inputs.conf)

_TCP_ROUTING = <tcpout_group_name>,<tcpout_group_name>,<tcpout_group_name>, ...
* Comma-separated list of tcpout group names.
* Using this, you can selectively forward the data to specific indexer(s).
* Specify the tcpout group the forwarder should use when forwarding the data.
  The tcpout group names are defined in outputs.conf with [tcpout:<tcpout_group_name>].
* Defaults to groups specified in "defaultGroup" in [tcpout] stanza in outputs.conf.
* To forward data from the "_internal" index, _TCP_ROUTING must explicitly be set to either "*" or
  a specific splunktcp target group.

UPDATE:

In order to do that, you'll need to clean out the fishbucket on the forwarding side. The fishbucket is where splunk keeps track of which files/events it has already read/forwarded.

http://splunk-base.splunk.com/answers/2954/how-can-i-re-index-all-the-data-in-my-environment

http://splunk-base.splunk.com/answers/2834/light-forwarder-syslog-fishbucket-problem

http://splunk-base.splunk.com/answers/46780/reset-splunkforwarder-to-re-read-file-from-beginning

Just be aware that if you do clean out the fishbucket on the forwarder, you'll get get some duplicate events, e.g. your file has events A, B, C and D, but your index contains only C and D (and you want to index A and B as well.) If you clean the fishbucket, you'll have A,B,C,C,D,D in your index. Depending on what you have in your index, you may also want to clean out the index on the indexer as well, giving you A,B,C,D in the index.

Hope this helps,

Kristian

View solution in original post

kristian_kolb
Ultra Champion

A couple of things,

If you call the file inputs.txt you will have problems, it should be called inputs.conf, otherwise Splunk will not recognize it. The same goes for outputs.conf

The order of precedence can be found in the docs.

You must configure the indexer to listen on port 9997. (manager -> forwarding and receiving -> configure receiving -> new).

Make sure that you have no firewalls in between (or local), blocking the traffic.

By the way, I don't think you'll need the _TCP_ROUTING stuff in the form you've written it;

(from the docs on inputs.conf)

_TCP_ROUTING = <tcpout_group_name>,<tcpout_group_name>,<tcpout_group_name>, ...
* Comma-separated list of tcpout group names.
* Using this, you can selectively forward the data to specific indexer(s).
* Specify the tcpout group the forwarder should use when forwarding the data.
  The tcpout group names are defined in outputs.conf with [tcpout:<tcpout_group_name>].
* Defaults to groups specified in "defaultGroup" in [tcpout] stanza in outputs.conf.
* To forward data from the "_internal" index, _TCP_ROUTING must explicitly be set to either "*" or
  a specific splunktcp target group.

UPDATE:

In order to do that, you'll need to clean out the fishbucket on the forwarding side. The fishbucket is where splunk keeps track of which files/events it has already read/forwarded.

http://splunk-base.splunk.com/answers/2954/how-can-i-re-index-all-the-data-in-my-environment

http://splunk-base.splunk.com/answers/2834/light-forwarder-syslog-fishbucket-problem

http://splunk-base.splunk.com/answers/46780/reset-splunkforwarder-to-re-read-file-from-beginning

Just be aware that if you do clean out the fishbucket on the forwarder, you'll get get some duplicate events, e.g. your file has events A, B, C and D, but your index contains only C and D (and you want to index A and B as well.) If you clean the fishbucket, you'll have A,B,C,C,D,D in your index. Depending on what you have in your index, you may also want to clean out the index on the indexer as well, giving you A,B,C,D in the index.

Hope this helps,

Kristian

kristian_kolb
Ultra Champion

see update above.

0 Karma

sieutruc
Contributor

It works now, but it just recorded in real time all events in the end of the file without the total file. How to catch up all the events in the beginning of the file ? I delete the option the followTail but nothing changes

0 Karma

kristian_kolb
Ultra Champion

Is your indexer listening on port 9997 as per the instructions above?

0 Karma

sieutruc
Contributor

Sorry for my wrong typing, all files have .conf extention, and all firewalls are turned off. But i still don't see it in example indexer. Is there a test for that ?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...