How can I build regex to extract data from unstructured Domain Time Logs?

[neltonk]

Hi,

Log files contain header and summary information in the beginning of the file. The number of header + summary lines in the log are random. The individual metrics are loaded after a label in the log after following format:

----------------------------------
INDIVIDUAL SYNCHRONIZATION RECORDS
----------------------------------


              Check Time      Time Source  Set  PhAdj   Check Reason  Seconds delta from Time Source (Report Source)
------------------------  ---------------  ---  -----  -------------  ---------------------------------------------------
Sun 04 Feb 2018 13:15:36     10.128.197.9   N     -16            PTP  +0000.0000017 (Full Client)
Sun 04 Feb 2018 13:15:37     10.128.197.9   N     +16            PTP   0000.0000000
Sun 04 Feb 2018 13:15:38     10.128.197.9   N     -16            PTP  -0000.0000156
Sun 04 Feb 2018 13:15:39     10.128.197.9   N       0            PTP  -0000.0000101
Sun 04 Feb 2018 13:15:40     10.128.197.9   N     +16            PTP  -0000.0000126
Sun 04 Feb 2018 13:15:41     10.128.197.9   N     -16            PTP  -0000.0000142
Sun 04 Feb 2018 13:15:42     10.128.197.9   N     +16            PTP  +0000.0000043
Sun 04 Feb 2018 13:15:43     10.128.197.9   N     +16            PTP  +0000.0000032
Sun 04 Feb 2018 13:15:44     10.128.197.9   N     -16            PTP  +0000.0000018
Sun 04 Feb 2018 13:15:45     10.128.197.9   N     +16            PTP  +0000.0000006
Sun 04 Feb 2018 13:15:46     10.128.197.9   N     -16            PTP  +0000.0000006
Sun 04 Feb 2018 13:15:47     10.128.197.9   N     +16            PTP  -0000.0000014
Sun 04 Feb 2018 13:15:48     10.128.197.9   N     -16            PTP  -0000.0000036

I need to pick the seconds delta. Is there a way to only send this data to splunk from the universal forwarder? Do we use the props.conf for this... please help. I am new to regex and need some help to build a regex for this.

Thanks

[richgalloway]

The universal forwarder does not use the props.conf file. Put the props settings on your indexer(s). If you must process the file locally, use a heavy forwarder.

[neltonk]

Thanks a lot for the quick response. however, is there any other way to retrieve the already indexed data(format as mentioned above) in Splunk Enterprise. Is it possible to give regex expression when searching to ignore all data before the section:

---

INDIVIDUAL SYNCHRONIZATION RECORDS

---

[cpetterborg]

You could do a rex command using sed mode and have it delete the data that doesn't have the format of the informational lines. That should be easy enough, but I'm not sure what you indexed data is looking like:

Is this data showing up in a search as a single event, a set of one-line events, or a mix of one- and multi-line events?

If you want to use the old data, then you will have to do search-time modification of the data that is returned from the search. For future data it is possible to remove the lines that don't start with a timestamp, leaving only the valuable data lines.