topic Re: why my sourcetype time extraction not take effect ? in Getting Data In

why my sourcetype time extraction not take effect ?

meg_li — Thu, 05 Mar 2020 08:40:52 GMT

first a log sample:
{"offset":44469279,"messages":"<190>Mar 5 2020 06:40:55 WH-USG-MAIN %%01POLICY/6/POLICYPERMIT(l):vsys=public, protocol=6, source-ip=172.16.174.2, source-port=9054, destination-ip=10.251.30.14, destination-port=443, time=2020/3/5 14:40:55, source-zone=dmz, destination-zone=trust, rule-name=GRE.\u0000","fields":{"service":"network-log"},"client_ip":"10.251.0.254","time":"2020-03-05 14:41:20","prospector":{"type":"log"},"source":"/data/network/logs/network/buffer.b5a015d0cd6da0203206d47dc21494bdb.log","@timestamp":"2020-03-05T06:41:20.000Z","beat":{"version":"6.2.4","hostname":"network-log-input","name":"network-log-input"}}

i want to extract ,"time":"2020-03-05 14:41:20" this part for my indexed time _time field

you can see my sourcetype config like blow:

but i can't get this time , still use the server local time for the _time field.

Re: why my sourcetype time extraction not take effect ?

gcusello — Wed, 30 Sep 2020 04:28:20 GMT

Hi @meg_li,
TIME_PREFIX accept a regex and " is a special char so you should try with this regex:

TIME_PREFIX  = \"time\":\"

Ciao.
Giuseppe

Re: why my sourcetype time extraction not take effect ?

to4kawa — Thu, 05 Mar 2020 08:49:51 GMT

TIMESTAMP_FIELDS=time

Re: why my sourcetype time extraction not take effect ?

meg_li — Thu, 05 Mar 2020 09:25:44 GMT

yes , i had try this TIME_PREFIX = \"time\":\" but still can't work.

Re: why my sourcetype time extraction not take effect ?

meg_li — Thu, 05 Mar 2020 09:25:59 GMT

yes , i had try this TIME_PREFIX = \"time\":\" but still can't work.

Re: why my sourcetype time extraction not take effect ?

gcusello — Thu, 05 Mar 2020 09:32:33 GMT

Hi @meg_li,
where do you put the props.conf file?
it must be on the Indexers or (when present) on Heavy Forwarders, not on Universal Forwarders.

Ciao.
Giuseppe

Re: why my sourcetype time extraction not take effect ?

meg_li — Thu, 05 Mar 2020 09:40:07 GMT

i have only one splunk server , and i only config it in web ui,Source Types page, not the props.conf file

Re: why my sourcetype time extraction not take effect ?

gcusello — Wed, 30 Sep 2020 04:28:22 GMT

Hi @meg_li,
all these configurations that you do by web gui are in a configuration file called props.conf that you can find in $SPLUNK_HOME/etc/system/local or in $SPLUNK_HOME/etc/apps/app_name/local.

I think that also the file source is in the same server.

If yes, try to make ingestion using the web gui procedure, in this way you can immediately see if Splunk read correctly or not the log.

Ciao.
Giuseppe

Re: why my sourcetype time extraction not take effect ?

meg_li — Thu, 05 Mar 2020 10:09:23 GMT

sorry ,i don't understand your meaning, i have not edit the props.conf file directly, only use the web ui, what i can do next , i still can't get the time field i want

Re: why my sourcetype time extraction not take effect ?

gcusello — Wed, 30 Sep 2020 04:28:25 GMT

Hi @meg_li,
don't worry, it was an explanation of what happens when you modify something using web gui.

As I said, use the guided ingestion procedure, so you can immediately test if the TIME_PREFIX and TIME_FORMAT is correct, you can find it at Settings.
Then choose the file to ingest and set the options TIME_PREFIX: in this way Splunk diplays the recognized timestamp and you can save the configuration in the used sourcetype.

Ciao.
Giuseppe

Re: why my sourcetype time extraction not take effect ?

meg_li — Thu, 05 Mar 2020 10:47:04 GMT

thanks, i had try to use this log sample in a file to go through the ingest procedure, also use the same sourcetype ,the _time is correct extracted.

but i was using udp receive the true log come in, the same sourcetype not work as i wish.

i don't know why .

Re: why my sourcetype time extraction not take effect ?

gcusello — Thu, 05 Mar 2020 10:49:02 GMT

Hi @meg_li,
try to configure your UDP input to use that sourcetype.

Ciao.
Giuseppe

Re: why my sourcetype time extraction not take effect ?

meg_li — Thu, 05 Mar 2020 10:51:10 GMT

it's the same sourcetype, i had created

Re: why my sourcetype time extraction not take effect ?

manjunathmeti — Wed, 30 Sep 2020 04:29:51 GMT

JSON is in valid format. You don't need to provide TIME_PREFIX and TIME_FORMAT. FIeld time is already extracted so you just need provide TIMESTAMP_FIELDS = time. This will set _time to time values.

Re: why my sourcetype time extraction not take effect ?

gcusello — Thu, 05 Mar 2020 11:01:06 GMT

Hi @meg_li,
see in the log wrong ingested what's the associated sourcetype and the log format, maybe it's different.

Ciao.
Giuseppe

Re: why my sourcetype time extraction not take effect ?

meg_li — Fri, 06 Mar 2020 01:36:09 GMT

thanks everyone , all the configuration is correct, at last ,i restart splunk service, now it is extract ok now .
:)