Automatic extraction of fields not happening for json data input to Splunk through udp

[sanchitguptaiit]

I have a process to send json format data to Splunk on an udp port. In settings I have mentioned sourcetype = _json. Splunk is able to detect and syntax highlight the data when search, but it is not able to automatically extract the fields. Any idea how to fix it?

Here is the inputs.conf

[udp://8704]
connection_host = dns
index = infra_mi6
sourcetype = _json

Sample search result, however all these fields are not showing up in Interesting Fields

4/6/16 
3:39:06.809 PM  
{ [-] 
    EventName:  SHEET_ACTIVATE 
    env:  PROD 
    host:  NB-9-1091 
    level:  INFO 
    msg:  null 
    ts:  2016-04-06T22:39:06.8099873Z 
    user:  sangupta 
    workbook:  Team Responsible - Low 
}
Show as raw text

[sanchitguptaiit]

I changed the source type to a new user defined and spunk automatically extracted all the events.

[IreneFdez]

What do you mean with 'source type to a new user defined'?

[sanchitguptaiit · Accepted solution]

thanks for the suggestions, i will try this out.

For now, i manually added extraction rules in props.conf

[svc_spldev@devpmsplunk1 local]$ cat props.conf
[_json]
EXTRACT-workbook = "workbook":"(?[^"]+)"
EXTRACT-env = "env":"(?[^"]+)"
EXTRACT-EventName = "EventName":"(?[^"]+)"
EXTRACT-hostname = "host":"(?[^"]+)"
EXTRACT-level = "level":"(?[^"]+)"
EXTRACT-msg = "msg":"(?[^"]+)"
EXTRACT-user = "user":"(?[^"]+)"

[twinspop]

In order for auto extraction with JSON to work, the entire event needs to be valid JSON. With UDP inputs, by default, splunk adds a timestamp. This makes the event no longer valid JSON. Add this to your UDP input stanzas:

no_appending_timestamp = true

Should fix it up.

EDIT: Apparently this isn't working. "Swing and a miss!"

[AlexandreB]

I have the exact same issue as the original poster and adding this stanzas in my udp input still do not automatically extract the data at index time.
As you mentionned, it does only appears when receiving data through UDP, and directly using a transmitter without going by UDP to send the same data will allow the fields to be automatically extracted.
Peculiar enough, using a manual non-referenced sourcetype for the UDP port does extract fields at index time, presumably by automatic lookup of the data by Splunk, regardless of if no_appending_timestamp is set. I would be interrested to know what further reasons would prevent the UDP configuration to to cater for extraction at index time.

[sanchitguptaiit]

Yes, this doesnt work for me either