topic Re: How to change _time data in All Apps and Add-ons

How to change _time data

vivekg72 — Mon, 10 Dec 2018 16:57:44 GMT

We inject Meinberg clock data into splunk one a day . We have observered that _time is filled when data is injected
Can we change this to use one field in data file

Day Sec GPS PPS IRIG NTP PTP FRQ STR

58461 00007 -0.000000004 0.000008975 ??? ??? ??? ??? ??? 20181209/00:00:07/UTC
58461 00020 0.000000000 0.000008870 ??? ??? ??? ??? ??? 20181209/00:00:20/UTC
58461 00033 -0.000000001 0.000009869 ??? ??? ??? ??? ??? 20181209/00:00:33/UTC

We want _time to be populated with STR data but injection data ..

Please advise how we can do this

Thanks

vivek

Re: How to change _time data

somesoni2 — Mon, 10 Dec 2018 17:07:59 GMT

Would suggest reading this
http://docs.splunk.com/Documentation/Splunk/7.2.1/Data/HowSplunkextractstimestamps

Re: How to change _time data

whrg — Tue, 29 Sep 2020 22:19:44 GMT

Hello @vivekg72!

Have a look at the page which @somesoni2 posted.

Also, have a look at Configure timestamp recognition.

Basically, you need to create a new sourcetype which tells Splunk how to extract the timestamp to build the _time field. When defining timestamp recognition, you need to take special care of TIME_PREFIX and TIME_FORMAT.

I presume you masked your data with "???" so my response might not be 100% accurate. Having the timestamp at the end of the event with no fixed string in front of it is a little tricky. (Ideally, the timestamp is at the very beginning of the event.) However, check this out:

[meinberg]
DATETIME_CONFIG = 
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TIME_FORMAT = %Y%m%d/%H:%M:%S/%Z
TIME_PREFIX = ([^\s]+\s+){9}
category = Custom
pulldown_type = true

You will need to increase the Lookahead if your events are longer than your example events.

Re: How to change _time data

neltonk — Tue, 29 Sep 2020 22:24:50 GMT

Hi
Thanks for your response.
We are absolutely new to Splunk, please help...

We have a 5 node splunk cluster - 3 indexers, 1 master and 1 search head. We have ingested the data from meinberg clocks as illustrated above by my colleague. The data is exactly as represented above, nothing is masked.

The data is updated at source once a day and splunk is picking up the _time value as the splunk ingestion time. We want to change this to pick the time in the last column.
Do we need to delete the existing data in Splunk for this change.

The inputs.conf for universal forwarder is :

[monitor://\\ldn\dfs01\PTPLogs\DomainTime\MeinbergClocks]
host_regex = Clocks\(\w+).mrs
disabled = false
sourcetype = clocktimesynclogs
index = indexclocktimesynclogs

Should we add your suggested config to the end of the inputs.conf:
DATETIME_CONFIG =
1. NO_BINARY_CHECK = true
2. SHOULD_LINEMERGE = false
3. TIME_FORMAT = %Y%m%d/%H:%M:%S/%Z
4. TIME_PREFIX = ([^\s]+\s+){9}
5. category = Custom
6. pulldown_type = true

Thanks a lot for your help.

Best regards,

Nelton

Re: How to change _time data

whrg — Mon, 17 Dec 2018 13:12:25 GMT

Hello Nelton,
You should add my suggested config (don't forget the first line/the stanza [meinberg] or similar) to props.conf on your Universal Forwarder as well as on your Heavy Forwarder/Indexer (whatever system comes after the Universal Forwarder).

You also need to edit the "sourcetype" line in your inputs.conf on your Universal Forwarder to match the stanza in props.conf:
sourcetype = meinberg

Don't forget to restart Splunk after making changes to configuration files.
You don't need to delete the existing data in Splunk. However, the existing data will keep the wrong timestamp.

Re: How to change _time data

neltonk — Mon, 17 Dec 2018 17:11:51 GMT

Hi,

I have tried to apply this in non-prod before applying in PROD. I have few questions... Please help.

As shown in the image, the rule is being applied to the first row with the headers, how to ignore the rule for the first row?
The date is being picked up however, it is adding 1 hour to the time. Now sure what is wrong here...

Thanks a lot for your help.

Best regards,

Nelton

Re: How to change _time data

neltonk — Tue, 18 Dec 2018 10:39:38 GMT

Could you please help...
Thanks

Re: How to change _time data

whrg — Tue, 29 Sep 2020 22:25:29 GMT

Hello Nelton,

Use the "Add comment" function when replying instead of posting a new answer.

Regarding your second question: Your logs contain a UTC timestamp. Splunk Web will display your local time zone. I guess for you that is GMT+1. (You can view/edit your user's time zone if you click on your username at the top and then click on "Preferences".)

Regarding your first question: Splunk can automatically extract the header and create the corresponding fields. See Extract fields from files with structured data. However, that only works for file-based inputs and for the "Upload" function in Splunk Web. Add the "FIELD_DELIMITER" and "HEADER_FIELD_DELIMITER" configuration:

https://ibb.co/3rjcBLV

Splunk complains that the header contains 9 fields whereas the other lines contain 10 fields.

You could also configure field extractions at search time instead at index time like shown here.