How do I delete all references of a host so it stops showing up on the host list?

[jradkowskiAAMC]

I've already deleted all references to the host in question in the internal indexes using the "| delete" search command. Additionally, I tried running the "clean" CLI command for both eventdata and globaldata and the host in question is still showing up!

I've confirmed the host is not sending any additional data to Splunk and would like to remove it completely. Is this possible without doing a "clean all" ?

[shookcapvm]

Does anyone have more specific information on where/how to run both the delete and clean commands? I am inexperienced with Splunk and am not sure what a lot of the forum answers are referring to. My end-goal is to remove one of my servers completely from the "Hosts" list. So far all I have done is uninstall the "Universal Splunk Forwarder" from the server I would like removed from Splunk.

[mntbighker]

Can I use globbing in props.conf ... [host::the_bad_host.*] ??

So far this has not worked for me:

props.conf:

[host::compute-0-*.local]
TRANSFORMS-nullhost = nullhost

transforms.conf:

[nullhost]
REGEX=.
DEST_KEY = queue
FORMAT = nullQueue

I am still getting stuff from compute-0-* appearing.

I also looked at this:

[root@ts-sl-server splunk]# /opt/splunk/bin/splunk cmd btool props list host
[host::compute*]
ANNOTATE_PUNCT = True
BREAK_ONLY_BEFORE =
BREAK_ONLY_BEFORE_DATE = True
CHARSET = UTF-8
DATETIME_CONFIG = /etc/datetime.xml
HEADER_MODE =
LEARN_SOURCETYPE = true
LINE_BREAKER_LOOKBEHIND = 100
MAX_DAYS_AGO = 2000
MAX_DAYS_HENCE = 2
MAX_DIFF_SECS_AGO = 3600
MAX_DIFF_SECS_HENCE = 604800
MAX_EVENTS = 256
MAX_TIMESTAMP_LOOKAHEAD = 128
MUST_BREAK_AFTER =
MUST_NOT_BREAK_AFTER =
MUST_NOT_BREAK_BEFORE =
SEGMENTATION = indexing
SEGMENTATION-all = full
SEGMENTATION-inner = inner
SEGMENTATION-outer = outer
SEGMENTATION-raw = none
SEGMENTATION-standard = standard
SHOULD_LINEMERGE = True
TRANSFORMS =
TRANSFORMS-nullhost = nullhost
TRUNCATE = 10000
maxDist = 100

It also occurs to me that the hosts I need to filter are the compute nodes in our clusters. So the forwarder is the head node. I'm not sure if the digester sees the forwarder as the host or if the fact the entries appear in Splunk with the hostname identified as the one I need to fileter should mean this should be working?

update: I hit myself in the head and moved the nullhost rule from the server to the forwarders (duh), and now I have another source of noise eliminated. The pie charts and data are actually becoming somewhat useful for the first time since I started using Splunk several years ago.

[jrodman]

My understanding is in modern versions of Splunk (late 4.0 and 4.1) the |delete should be eventually sufficient, but there may be a lag until the index-wide data is recaulculated and this lag is inversely proportional to the incoming data rate.

[the_wolverine · Accepted solution]

Once you've cleaned up the existing data, you can set up a transform to route all its events to nullQueue in the event it decides to send again.

props.conf:

[host::the_bad_host]
TRANSFORMS-nullhost = nullhost

transforms.conf:

[nullhost]
REGEX=.
DEST_KEY = queue
FORMAT = nullQueue

[lukeh]

Replace queue = nullQueue with FORMAT = nullQueue

[thall79]

You could tag the server as decommissioned (tag=decom) and in your searches exclude that tag.

Travis.

[jradkowskiAAMC]

Looks as if I asked too soon!

I still had an input active on another host that was forcing itself to report as the host I wanted removed. Once I resolved that issue (issued an update via the Deployment Manager) and then did a clean on eventdata + globaldata I was able to start over again with the host missing.

I am still open to ideas on how to remove a host completely from Splunk for future use as I'm sure it will happen