topic Re: Replace space in timestamp with a 0 (with datetime.xml possibly?) in Getting Data In

Replace space in timestamp with a 0 (with datetime.xml possibly?) - Whitespace Padded Microseconds

pestatp — Thu, 04 Nov 2021 13:28:57 GMT

I have events that unfortunately use a space instead of a 0 in their timestamp field. The timestamp goes down to 6 decimal places, so there can be as many as 5 leading spaces in the decimal seconds section. Each event starts with the timestamp as below. As you can see, it has a leading space and I'd like to change that to a 0

[12-02-2020:08.31.44. 15133] SIP IN:

I have tried using SEDCMD in the props.conf, but it didn't seem to work on the newly indexed events, is my regex not correct or am I way off?

[sip_sbc]
SEDCMD-replace_space=s/^(\[[0-9-:\.]{20}\]) ()/\10\2/g

Edit: This is my current props.conf settings. This works fine for the timestamps that have 6 digits after the . but any of them that have leading spaces fail to get the proper timestamp

[sip_sbc] BREAK_ONLY_BEFORE_DATE = DATETIME_CONFIG = LINE_BREAKER = ----------------------------------------------------------------------------------------([\r\n]+) NO_BINARY_CHECK = true SHOULD_LINEMERGE = false TIME_FORMAT = %m-%d-%Y:%H.%M.%S.%6N TIME_PREFIX = [ category = Custom pulldown_type = 1 BREAK_ONLY_BEFORE_TIME = disabled = false MUST_BREAK_AFTER =

Edit No. 2: I have added a custom datetime.xml file for the app. It does have an effect, but it's not working quite right. It doesn't pad the leading spaces to 0, it just removes the space and therefor causes the subseconds to be much higher than they are supposed to be on some timestamps.

This datetime.xml caused an event with the timestamp: 01-08-2021:11.28.23. 8213 (note the 2 spaces before 8213) to be parsed as 1/8/21 11:28:23.821 (this is 8 10ths of a second after the timestamp should be)

It should be 1/8/21 11:28:23.008213

Re: Replace space in timestamp with a 0

richgalloway — Wed, 02 Dec 2020 15:53:08 GMT

Can you ignore the microseconds part of the timestamp?

IIRC, SEDCMD executes after timestamp extraction so there's nothing it can do to a timestamp.

Perhaps Cribl can do the job.

Re: Replace space in timestamp with a 0

pestatp — Wed, 02 Dec 2020 15:55:49 GMT

No, I cannot ignore it. These are SIP messages and they need to be in as precise order as possible to ensure the call flow is correct. Many of the messages are within just a few thousandths of a second of each other.

Re: Replace space in timestamp with a 0

pestatp — Fri, 08 Jan 2021 14:30:53 GMT

After some more searching, I came across datetime.xml. It looks like that can be used to do custom datetime extraction although, I am a bit confused by the documentation. Does anyone know if that would work for this scenario?

Re: Replace space in timestamp with a 0

richgalloway — Fri, 08 Jan 2021 15:25:50 GMT

Yes, the datetime.xml file can be customized. It should be pretty straightforward to add support for 9 decimal places. Look for where subseconds are extracted from the pattern that matches your time format.

Re: Replace space in timestamp with a 0 (with datetime.xml possibly?)

scelikok — Fri, 08 Jan 2021 15:57:18 GMT

Hi @pestatp,

Can you try below datetime.xml?

If this reply helps you an upvote is appreciated.

Re: Replace space in timestamp with a 0 (with datetime.xml possibly?)

pestatp — Fri, 08 Jan 2021 16:39:47 GMT

@scelikok

That is the closest I have gotten it so far, but it doesn't look like it's treating the spaces as zeros, it's just ignoring them.

It indexed 01-08-2021:11.28.23. 8213 as 1/8/21 11:28:23.821 AM - that is better than what it was doing before, but still not quite right. It should be indexed as 1/8/21 11:28:23.008213 AM

Re: Replace space in timestamp with a 0 (with datetime.xml possibly?)

pestatp — Thu, 04 Nov 2021 12:54:36 GMT

I have still not been able to figure this out. I haven't seen anything in newer versions that would help with this, anyone know of anything?