Getting Data In

why my sourcetype time extraction not take effect ?

meg_li
New Member

first a log sample:
{"offset":44469279,"messages":"<190>Mar 5 2020 06:40:55 WH-USG-MAIN %%01POLICY/6/POLICYPERMIT(l):vsys=public, protocol=6, source-ip=172.16.174.2, source-port=9054, destination-ip=10.251.30.14, destination-port=443, time=2020/3/5 14:40:55, source-zone=dmz, destination-zone=trust, rule-name=GRE.\u0000","fields":{"service":"network-log"},"client_ip":"10.251.0.254","time":"2020-03-05 14:41:20","prospector":{"type":"log"},"source":"/data/network/logs/network/buffer.b5a015d0cd6da0203206d47dc21494bdb.log","@timestamp":"2020-03-05T06:41:20.000Z","beat":{"version":"6.2.4","hostname":"network-log-input","name":"network-log-input"}}

i want to extract ,"time":"2020-03-05 14:41:20" this part for my indexed time _time field

you can see my sourcetype config like blow:
alt text

but i can't get this time , still use the server local time for the _time field.

0 Karma

meg_li
New Member

thanks everyone , all the configuration is correct, at last ,i restart splunk service, now it is extract ok now .
:)

0 Karma

manjunathmeti
Champion

JSON is in valid format. You don't need to provide TIME_PREFIX and TIME_FORMAT. FIeld time is already extracted so you just need provide TIMESTAMP_FIELDS = time. This will set _time to time values.

alt text

0 Karma

meg_li
New Member

yes , i had try this TIME_PREFIX = \"time\":\" but still can't work.

0 Karma

to4kawa
SplunkTrust
SplunkTrust

TIMESTAMP_FIELDS=time

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @meg_li,
TIME_PREFIX accept a regex and " is a special char so you should try with this regex:

TIME_PREFIX  = \"time\":\"

Ciao.
Giuseppe

0 Karma

meg_li
New Member

i have only one splunk server , and i only config it in web ui,Source Types page, not the props.conf file

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @meg_li,
all these configurations that you do by web gui are in a configuration file called props.conf that you can find in $SPLUNK_HOME/etc/system/local or in $SPLUNK_HOME/etc/apps/app_name/local.

I think that also the file source is in the same server.

If yes, try to make ingestion using the web gui procedure, in this way you can immediately see if Splunk read correctly or not the log.

Ciao.
Giuseppe

0 Karma

meg_li
New Member

sorry ,i don't understand your meaning, i have not edit the props.conf file directly, only use the web ui, what i can do next , i still can't get the time field i want

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @meg_li,
don't worry, it was an explanation of what happens when you modify something using web gui.

As I said, use the guided ingestion procedure, so you can immediately test if the TIME_PREFIX and TIME_FORMAT is correct, you can find it at Settings.
Then choose the file to ingest and set the options TIME_PREFIX: in this way Splunk diplays the recognized timestamp and you can save the configuration in the used sourcetype.

Ciao.
Giuseppe

0 Karma

meg_li
New Member

thanks, i had try to use this log sample in a file to go through the ingest procedure, also use the same sourcetype ,the _time is correct extracted.

but i was using udp receive the true log come in, the same sourcetype not work as i wish.

i don't know why .

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @meg_li,
try to configure your UDP input to use that sourcetype.

Ciao.
Giuseppe

0 Karma

meg_li
New Member

it's the same sourcetype, i had created

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @meg_li,
see in the log wrong ingested what's the associated sourcetype and the log format, maybe it's different.

Ciao.
Giuseppe

0 Karma

meg_li
New Member

yes , i had try this TIME_PREFIX = \"time\":\" but still can't work.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @meg_li,
where do you put the props.conf file?
it must be on the Indexers or (when present) on Heavy Forwarders, not on Universal Forwarders.

Ciao.
Giuseppe

0 Karma