Getting Data In

Why is the UF version on forwarder different than what the indexer is seeing?

MrMcGeough
Explorer

I recently upgraded all of my Universal Forwarders (UF) to 7.0.3 from various version levels (some 6.3.3, some were 7.x).

On one of the forwarders (AIX) when I run the command;
./splunk version
I get; "Splunk Universal Forwarder 7.0.3"

But a search to list forwarder versions on the indexer lists a different version for the same host;

index=_internal sourcetype=splunkd group=tcpin_connections | stats first(version) by hostname
I get; "Version 7.0.2"

Why are the versions being listed differently? After the upgrade, this is the only UF not listing 7.0.3 at the indexer.

0 Karma
1 Solution

jlvix1
Communicator

Pardon my negativity, it would not surprise me if this is a bug, surely the wrong version is being sent by the forwarder.

I would recommend uninstalling and reinstalling, if the same then obviously a defect!

View solution in original post

joesrepsol
Path Finder

Same experience here. I had v6.5.0 and upgraded to v7.1.0 forwarder and 10% of them from linux/AIX are showing the old version still installed. I check on each VM... they are v7.1.0. I've restarted them. Still no change. Hard to tell which of the v6.5.0's showing up in my deployment are truly not running on v7.1.0 or not. Frustrating.

0 Karma

ckurtz
Path Finder

This looks like it may be a bug in the AIX build. I just tried the linux tgz and Mac dmg, and they both look right.

I suggest opening a support case, it may have just been a bad build of the AIX UF.

0 Karma

MrMcGeough
Explorer

Thanks for your comment ckurtz.

I have 6 AIX forwarders and all were updated at the same time. Only one of them is mis-reporting the version.

If I had access to do the install myself I would, but they have the servers so locked down here I have to open a change ticket which takes about 1/2 hour, wait for assessment by many people, and then wait a week to have the steps performed.

0 Karma

ckurtz
Path Finder

That's really odd! Well, I don't think it matters much, but it does sound like a bad install, maybe something didn't get properly overwritten. If you can request that the Ops Team reinstall 7.0.3 on that machine is might not be a bad idea. I don't think you're in any danger of Bad Things.

0 Karma

jlvix1
Communicator

Pardon my negativity, it would not surprise me if this is a bug, surely the wrong version is being sent by the forwarder.

I would recommend uninstalling and reinstalling, if the same then obviously a defect!

MrMcGeough
Explorer

Sounds more realistic than negative to me.

0 Karma

MrMcGeough
Explorer

Fixed! After backing up /opt/splunkforwarder/etc/auth/mycerts and /opt/splunkforwarder/etc/system/local I went ahead and got the Ops group to delete /opt/splunkforwarder/etc (and all subs). Uploaded a new copy of the install tar for 7.0.3 and had them do a gunzip to install.

On startup the server.pem file was missing for some reason so I grabbed a working copy from another 7.0.3 host and it worked.

The indexer is now reporting 7.0.3 for this forwarder properly!

p_gurav
Champion

Can you try | stats latest(version) instead of first.

0 Karma

MrMcGeough
Explorer

Thanks for your reply. Same result I'm afraid. The Monitoring Console also reports version 7.0.2 for this host. I think that's how it's coming to the indexer. I just don't know why.

0 Karma

p_gurav
Champion
0 Karma

MrMcGeough
Explorer

Have tried that, but I just ran it again. No change.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...