How to change _time data

[vivekg72]

We inject Meinberg clock data into splunk one a day . We have observered that _time is filled when data is injected
Can we change this to use one field in data file

Day Sec GPS PPS IRIG NTP PTP FRQ STR

58461 00007 -0.000000004 0.000008975 ??? ??? ??? ??? ??? 20181209/00:00:07/UTC
58461 00020 0.000000000 0.000008870 ??? ??? ??? ??? ??? 20181209/00:00:20/UTC
58461 00033 -0.000000001 0.000009869 ??? ??? ??? ??? ??? 20181209/00:00:33/UTC

We want _time to be populated with STR data but injection data ..

Please advise how we can do this

Thanks

vivek

[neltonk]

Could you please help...
Thanks

[neltonk]

Hi,

I have tried to apply this in non-prod before applying in PROD. I have few questions... Please help.

1. As shown in the image, the rule is being applied to the first row with the headers, how to ignore the rule for the first row?
2. The date is being picked up however, it is adding 1 hour to the time. Now sure what is wrong here...

Thanks a lot for your help.

Best regards,

Nelton

[whrg]

Hello Nelton,

Use the "Add comment" function when replying instead of posting a new answer.

Regarding your second question: Your logs contain a UTC timestamp. Splunk Web will display your local time zone. I guess for you that is GMT+1. (You can view/edit your user's time zone if you click on your username at the top and then click on "Preferences".)

Regarding your first question: Splunk can automatically extract the header and create the corresponding fields. See Extract fields from files with structured data. However, that only works for file-based inputs and for the "Upload" function in Splunk Web. Add the "FIELD_DELIMITER" and "HEADER_FIELD_DELIMITER" configuration:

https://ibb.co/3rjcBLV

Splunk complains that the header contains 9 fields whereas the other lines contain 10 fields.

You could also configure field extractions at search time instead at index time like shown here.

[neltonk]

Hi
Thanks for your response.
We are absolutely new to Splunk, please help...

We have a 5 node splunk cluster - 3 indexers, 1 master and 1 search head. We have ingested the data from meinberg clocks as illustrated above by my colleague. The data is exactly as represented above, nothing is masked.

The data is updated at source once a day and splunk is picking up the _time value as the splunk ingestion time. We want to change this to pick the time in the last column.
Do we need to delete the existing data in Splunk for this change.

The inputs.conf for universal forwarder is :

[monitor://\\ldn\dfs01\PTPLogs\DomainTime\MeinbergClocks]
host_regex = Clocks\(\w+).mrs
disabled = false
sourcetype = clocktimesynclogs
index = indexclocktimesynclogs

Should we add your suggested config to the end of the inputs.conf:
DATETIME_CONFIG =
1. NO_BINARY_CHECK = true
2. SHOULD_LINEMERGE = false
3. TIME_FORMAT = %Y%m%d/%H:%M:%S/%Z
4. TIME_PREFIX = ([^\s]+\s+){9}
5. category = Custom
6. pulldown_type = true

Thanks a lot for your help.

Best regards,

Nelton

[whrg]

Hello Nelton,
You should add my suggested config (don't forget the first line/the stanza [meinberg] or similar) to props.conf on your Universal Forwarder as well as on your Heavy Forwarder/Indexer (whatever system comes after the Universal Forwarder).

You also need to edit the "sourcetype" line in your inputs.conf on your Universal Forwarder to match the stanza in props.conf:
sourcetype = meinberg

Don't forget to restart Splunk after making changes to configuration files.
You don't need to delete the existing data in Splunk. However, the existing data will keep the wrong timestamp.

[whrg]

Hello @vivekg72!

Have a look at the page which @somesoni2 posted.

Also, have a look at Configure timestamp recognition.

Basically, you need to create a new sourcetype which tells Splunk how to extract the timestamp to build the _time field. When defining timestamp recognition, you need to take special care of TIME_PREFIX and TIME_FORMAT.

I presume you masked your data with "???" so my response might not be 100% accurate. Having the timestamp at the end of the event with no fixed string in front of it is a little tricky. (Ideally, the timestamp is at the very beginning of the event.) However, check this out:

[meinberg]
DATETIME_CONFIG = 
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TIME_FORMAT = %Y%m%d/%H:%M:%S/%Z
TIME_PREFIX = ([^\s]+\s+){9}
category = Custom
pulldown_type = true

You will need to increase the Lookahead if your events are longer than your example events.

[somesoni2]

Would suggest reading this
http://docs.splunk.com/Documentation/Splunk/7.2.1/Data/HowSplunkextractstimestamps