Why is the UF version on forwarder different than what the indexer is seeing?

[MrMcGeough]

I recently upgraded all of my Universal Forwarders (UF) to 7.0.3 from various version levels (some 6.3.3, some were 7.x).

On one of the forwarders (AIX) when I run the command;
./splunk version
I get; "Splunk Universal Forwarder 7.0.3"

But a search to list forwarder versions on the indexer lists a different version for the same host;

index=_internal sourcetype=splunkd group=tcpin_connections | stats first(version) by hostname
I get; "Version 7.0.2"

Why are the versions being listed differently? After the upgrade, this is the only UF not listing 7.0.3 at the indexer.

[joesrepsol]

Same experience here. I had v6.5.0 and upgraded to v7.1.0 forwarder and 10% of them from linux/AIX are showing the old version still installed. I check on each VM... they are v7.1.0. I've restarted them. Still no change. Hard to tell which of the v6.5.0's showing up in my deployment are truly not running on v7.1.0 or not. Frustrating.

[ckurtz]

This looks like it may be a bug in the AIX build. I just tried the linux tgz and Mac dmg, and they both look right.

I suggest opening a support case, it may have just been a bad build of the AIX UF.

[MrMcGeough]

Thanks for your comment ckurtz.

I have 6 AIX forwarders and all were updated at the same time. Only one of them is mis-reporting the version.

If I had access to do the install myself I would, but they have the servers so locked down here I have to open a change ticket which takes about 1/2 hour, wait for assessment by many people, and then wait a week to have the steps performed.

[ckurtz]

That's really odd! Well, I don't think it matters much, but it does sound like a bad install, maybe something didn't get properly overwritten. If you can request that the Ops Team reinstall 7.0.3 on that machine is might not be a bad idea. I don't think you're in any danger of Bad Things.

[jlvix1 · Accepted solution]

Pardon my negativity, it would not surprise me if this is a bug, surely the wrong version is being sent by the forwarder.

I would recommend uninstalling and reinstalling, if the same then obviously a defect!

[MrMcGeough]

Sounds more realistic than negative to me.

[MrMcGeough]

Fixed! After backing up /opt/splunkforwarder/etc/auth/mycerts and /opt/splunkforwarder/etc/system/local I went ahead and got the Ops group to delete /opt/splunkforwarder/etc (and all subs). Uploaded a new copy of the install tar for 7.0.3 and had them do a gunzip to install.

On startup the server.pem file was missing for some reason so I grabbed a working copy from another 7.0.3 host and it worked.

The indexer is now reporting 7.0.3 for this forwarder properly!

[p_gurav]

Can you try | stats latest(version) instead of first.

[MrMcGeough]

Thanks for your reply. Same result I'm afraid. The Monitoring Console also reports version 7.0.2 for this host. I think that's how it's coming to the indexer. I just don't know why.

[p_gurav]

Can you try rebuilding forwarder asset table. Refer below doc:
https://docs.splunk.com/Documentation/Splunk/7.0.3/DMC/Configureforwardermonitoring#Rebuild_the_forw...

[MrMcGeough]

Have tried that, but I just ran it again. No change.