Getting Data In

Duplicate host field after indexing JSON event

ColinCH
Path Finder

We are indexing a Logfile that has the following JSON format:

{"_check_command":"hostalive",_execution_time":4.0079541206359863281,**,"host":"myHostname",**"short_message":"PING OK","timestamp":1502959941.4374480247}

The Splunkforwarder is on the same server with the same "host" entry in the

inputs.conf 
[default] 
host = myHostname

When i'm searching in Splunk after the events from the Logfile, i have the problem that every Events has 2 Host fields, that one that is extracted from the JSON and that from the Forwarder. The counter is also doubled)

My first opinion was i cut out the host field in the JSON but it's not my prefered one.

Any ideas?

EDIT:
I added following line to the props.conf:
SEDCMD = s/\"host\"/\"hostname\"/

Tags (2)
0 Karma
1 Solution

cpetterborg
SplunkTrust
SplunkTrust

I'd modify the source at index time to contain hostname instead of host so that there would be no conflict. That will happen so that any search time field extractions will keep the host names separate. It is hard to deal with the host having two values if it does automatic field extractions.

The other thing that you could do at search time it use a rex to create a hostname field, then eliminate the extra host value that matches the hostname in the event, something like this:

| makeresults 
| eval host="orighost,myHostname" 
| makemv host delim="," 
| eval _raw="{\"_check_command\":\"hostalive\",_execution_time\":4.0079541206359863281,**,\"host\":\"myHostname\",**\"short_message\":\"PING OK\",\"timestamp\":1502959941.4374480247}" 
| rex "\"host\":\"(?P<hostname>[^\"]*)\"" 
| mvexpand host 
| where host!=hostname

The first four lines are just setting up the data, the last three are doing the work of setting the host and hostname properly.

View solution in original post

0 Karma

cpetterborg
SplunkTrust
SplunkTrust

I'd modify the source at index time to contain hostname instead of host so that there would be no conflict. That will happen so that any search time field extractions will keep the host names separate. It is hard to deal with the host having two values if it does automatic field extractions.

The other thing that you could do at search time it use a rex to create a hostname field, then eliminate the extra host value that matches the hostname in the event, something like this:

| makeresults 
| eval host="orighost,myHostname" 
| makemv host delim="," 
| eval _raw="{\"_check_command\":\"hostalive\",_execution_time\":4.0079541206359863281,**,\"host\":\"myHostname\",**\"short_message\":\"PING OK\",\"timestamp\":1502959941.4374480247}" 
| rex "\"host\":\"(?P<hostname>[^\"]*)\"" 
| mvexpand host 
| where host!=hostname

The first four lines are just setting up the data, the last three are doing the work of setting the host and hostname properly.

0 Karma

ColinCH
Path Finder

ahh, thank you. I did not think of that. I will SED the host field to hostname, so i don't need do delete it!

0 Karma

cpetterborg
SplunkTrust
SplunkTrust

If this works for you, please accept the answer so that future viewers will know that it is a valid answer to your question. 🙂 Thanks!

0 Karma
Get Updates on the Splunk Community!

Splunk Enterprise Security: Your Command Center for PCI DSS Compliance

Every security professional knows the drill. The PCI DSS audit is approaching, and suddenly everyone's asking ...

Developer Spotlight with Guilhem Marchand

From Splunk Engineer to Founder: The Journey Behind TrackMe    After spending over 12 years working full time ...

Cisco Catalyst Center Meets Splunk ITSI: From 'Payments Are Down' to Root Cause in ...

The Problem: When Networks and Services Don't Talk Payment systems fail at a retail location. Customers are ...