topic Re: Universal Forwarder not displaying data on SplunkWeb on another server in Getting Data In

Universal Forwarder not displaying data on SplunkWeb on another server

atewari — Fri, 30 Nov 2012 02:34:29 GMT

We have two Linux servers using Splunk 5.0.1 on 64-bit.

A full Splunk install (SplunkD and SplunkWeb). We created a Receive data port 8001 on this.
Universal Forwarder on second Linus server. We added a forwarder using command
```
splunk add forward-server server1:8001 -auth admin:somepassword
```

It was successfully added. We restarted the forwarder on server2. So we ran the command





splunk list forward-server

This command showed that server1:8001 was added but was not active. When we ran the list command again it said the file was locked. The metrics.log file says it is connected successfully.

But how do we view server2 data on SplunkWeb running on server1? We added the *nix App, but it cannot see server2 selection anywhere. We only see server1 info. Is there another step to activate the forwarder on server2 and enable something on server1 to view server2 logs?

The "deploy" forwarder documentation is confusing. It gives a few commands and then asks to test the deployment of the forwarder without instructions on what to test.

Can anyone point us to the next steps - links, answers, anything?

Thanks

Re: Universal Forwarder not displaying data on SplunkWeb on another server

atewari — Fri, 30 Nov 2012 08:03:37 GMT

We cannot see the host server that runs the universal forwarder on the summary page on the right side. On server1 (SplunkWeb full install), we changed the local/inputs.conf under [default] stanza and changed the host = it then showed two servers. But it still collects the main server1 data only.

Is there anything else we need to do? We basically changed the outputs.conf on server2 (with universal forwarder) and the inputs.conf (full install). We know that connection is established because the metrics.log shows receiving data,

Re: Universal Forwarder not displaying data on SplunkWeb on another server

Drainy — Fri, 30 Nov 2012 08:54:00 GMT

Ok, lets rewind a little. Could you post your inputs.conf from the indexer and the outputs.conf from the forwarder?

Something worth remembering, you need to define inputs (in inputs.conf) before the forwarder will send anything, also if you define a different index to default in outputs.conf on the forwarder then the summary page won't show any detail from it.

Re: Universal Forwarder not displaying data on SplunkWeb on another server

aneeshkjm123 — Fri, 30 Nov 2012 09:49:54 GMT

Check your port 8001 in the forwarder is blocked. Disable firewall and try if you are not able to unblock the port

Re: Universal Forwarder not displaying data on SplunkWeb on another server

atewari — Mon, 28 Sep 2020 12:53:32 GMT

Thanks Drainy! Here is a complete rundown of the conf files.


Universal Forwarder on uf.xyz.com:8001

local/inputs.conf
==================
[default]
host = uf.xyz.com

local/outputs.conf
==================
[tcpout]
defaultGroup = default-autolb-group
disabled = false

[tcpout:default-autolb-group]
server = fullinstall.xyz.com:8001

[tcpout-server://fullinstall.xyz.com:8001]

Now, the files for splunk full install





Splunk full install fullinstall.xyz.com:8001

local/inputs.conf

[default]
host = fullinstall.xyz.com
disabled = 0
index = summary

[tcp://uf.xyz.com:8001]
disabled = 0
index = summary

local/outputs.conf

[tcpout]
indexAndForward = 0

Here is the splunkd.log on uf.xyz.com. It does connect to fullinstall.xyz.com and the uf.xyz.com also sends data to it





11-30-2012 02:00:49.012 -0600 INFO  TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk/...stash_new.

11-30-2012 02:00:49.012 -0600 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/etc/splunk.version.

11-30-2012 02:00:49.012 -0600 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk.

11-30-2012 02:00:49.012 -0600 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log.

11-30-2012 02:00:49.012 -0600 INFO  BatchReader - State transitioning from 2 to 0 (initOrResume).

11-30-2012 02:00:49.028 -0600 INFO  TcpOutputProc - Connected to idx=xxx.xxx.xxx.xxx:8001

Any suggestions would be much appreciated.

Thanks!

Re: Universal Forwarder not displaying data on SplunkWeb on another server

Drainy — Fri, 30 Nov 2012 14:17:02 GMT

I don't think you need that tcpout-server, but thats beside the point. Why are you sending it all into index=summary? delete that and let it drop into the default main index, the search app summary page won't show anything from indexes other than main.

As another test, try doing a search for index=summary and see if your data appears.

Re: Universal Forwarder not displaying data on SplunkWeb on another server

atewari — Fri, 30 Nov 2012 14:19:11 GMT

Thanks! We have confirmed that the forwarder has established connection and is sending data to the fullinstall server1. As Drainy pointed out, we may have left some configuration out. But we are not sure which stanza is missing to display the the universal forwarder host on the summary page.

Any suggestions? We have posted our conf files below.

Re: Universal Forwarder not displaying data on SplunkWeb on another server

DaveSavage — Fri, 30 Nov 2012 14:22:57 GMT

Atewari - note Drainy's comment here re different index in outputs.conf. You currently don't have that in the rundown below. Note especially that *nix uses the index 'os' and all canned searches are pre-disposed to that source!
Br and good luck. Dave

Re: Universal Forwarder not displaying data on SplunkWeb on another server

atewari — Fri, 30 Nov 2012 14:46:26 GMT

Drainy,
Thanks for your quick response. Here is what we now have in the inputs.conf on fulliinstall.xyz.com





[default]

host = fullinstall.xyz.com

disabled = 0

[tcp://uf.xyz.com:8001]
disabled = 0

No difference. could it be that all logs are sent to the same server and therefore two hosts are not shown.

We changed the inputs.conf [default] stanza to

host = anotherserver.xyz.com

When we did this, it showed this new host. But it collects data for the fullinstall.xyz.com

Are we missing anything else?

Re: Universal Forwarder not displaying data on SplunkWeb on another server

atewari — Fri, 30 Nov 2012 14:47:35 GMT

BTW, index=summary did not show any data.

Re: Universal Forwarder not displaying data on SplunkWeb on another server

atewari — Fri, 30 Nov 2012 14:48:01 GMT

BTW, index=summary did not show any data.

Re: Universal Forwarder not displaying data on SplunkWeb on another server

Drainy — Fri, 30 Nov 2012 14:49:15 GMT

Well, did you try the search for index=summary? Once your data has been forwarded/indexed it won't send them again unless you do a few cleanup tasks. Also that should be splunktcp:// instead of tcp:// in your inputs on the indexer.

Re: Universal Forwarder not displaying data on SplunkWeb on another server

atewari — Fri, 30 Nov 2012 15:32:03 GMT

BTW, index=summary did not show any data. I also changed the tcp to splunktcp

[default]
host = fullinstall.xyz.com
disabled = 0


[splunktcp://uf.xyz.com:8001] disabled = 0

I get the following info in splunkd.log on the forwarder server

11-30-2012 09:26:16.412 -0600 INFO  loader - Using cipher suite ALL:!aNULL:!eNULL:!LOW:!EXP:RC4+RSA:+HIGH:+MEDIUM
11-30-2012 09:26:16.504 -0600 INFO  TailingProcessor - TailWatcher initializing...
11-30-2012 09:26:16.505 -0600 INFO  TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk.
11-30-2012 09:26:16.505 -0600 INFO  TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk/...stash_new.
11-30-2012 09:26:16.505 -0600 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/etc/splunk.version.
11-30-2012 09:26:16.505 -0600 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk.
11-30-2012 09:26:16.505 -0600 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log.
11-30-2012 09:26:16.505 -0600 INFO  BatchReader - State transitioning from 2 to 0 (initOrResume).
11-30-2012 09:26:16.525 -0600 INFO  TcpOutputProc - Connected to idx=xxx.xxx.xxx.xxx:8001
11-30-2012 09:26:46.050 -0600 INFO  CMConfig - A splunktcp forwarder port is not configured in inputs.conf

In any case should I not see raw data if I used tcp instead of SplunkTCP? What does the splunktcp forwarder port not configured mean? Are there missing stanzas in the input.conf of forwrader server or fullinstall server?

Re: Universal Forwarder not displaying data on SplunkWeb on another server

DaveSavage — Fri, 30 Nov 2012 17:24:01 GMT

Drainy and Atewari - I stuck my hand up earlier on a feature of star-nix but not sure if you clocked it, or didn't need it...no worries either way. You said 'We added the star-nix App", but it cannot see server2 selection anywhere'....well, tbh you won't until you get the index 'os' sorted out. Traffic to *nix is expected in that space.
Cheers, Dave

Re: Universal Forwarder not displaying data on SplunkWeb on another server

atewari — Fri, 30 Nov 2012 17:31:57 GMT

Dave,
Do you see any issues with conf file changes we made for Splunk not to display both servers on summary page? I have increased the debugging to determine what is causing the forwarder server host not showing up on summary page of the portal.

Any insight would be greatly appreciated, thanks

Re: Universal Forwarder not displaying data on SplunkWeb on another server

DaveSavage — Fri, 30 Nov 2012 18:11:36 GMT

Atewari - not wishing to tread on Drainy's toes 'cos he is a good 'un and very solid....my two-pennneth relates purely to that of experience with the star-nix (am still trying to work out the tagging variables here) plug-in. Level playing field - you say summary page - by that you mean star-nix's...or Splunk's Search page? Portal being standard Splunk access?? We don't tend to use that word too much around here, albeit it will be a portal for someone, being web access.

Re: Universal Forwarder not displaying data on SplunkWeb on another server

DaveSavage — Fri, 30 Nov 2012 18:11:54 GMT

Your 2nd server with the forwarder on it needs to be firing at index 'os'...that's all. If its only those 2 servers then 9997 is fine, but we segregated out star-nix traffic from Windows, sending the former to (say) port 9996, then told the indexer's Data Inputs params (see Manager if you prefer GUI) under 'More Settings' to populate the index 'os'. We work in a highly secure environment complying to auditable Government levels, so need both.

Re: Universal Forwarder not displaying data on SplunkWeb on another server

atewari — Fri, 30 Nov 2012 18:37:23 GMT

Thanks Dave. At this point I am not even looking at *-nix. I am looking at Spunk's summary page. I added the index = os in the inputs.conf file and restarted both forwarder and splunkWeb. It did not do anything. Reviewing the log files, I know data is being transferred from server2 (univ. forwarder) to server1 (fullinstall).

I followed all instructions in the inputs.conf spec with no luck.

Splunk has great features, and we were hoping for an easier configuration. Any last suggestions that you think we can try?

thanks again for your help! We really appreciate it.

Re: Universal Forwarder not displaying data on SplunkWeb on another server

Drainy — Fri, 30 Nov 2012 22:20:29 GMT

Yeah, since atewari's inputs didn't contain anything to do with the *nix app nor did they mention it I skipped over it, a valid point though as it would show the same symptoms.

Re: Universal Forwarder not displaying data on SplunkWeb on another server

Drainy — Fri, 30 Nov 2012 22:22:13 GMT

D'oh. You aren't monitoring any files on the forwarder.... so it hasn't got anything to send, unless you're holding out on me here? 🙂 You need to add a monitor statement to your inputs.conf on the forwarder for it to actually monitor anything.