How to set INGEST

[noott211]

There is no time field in my log and I tried to get time from the source file name

I tried the settings below

myfile /var/log/data_01_20220507

/var/log/data_02_20220506

.

.

transforms.conf

[get_date]

SOURCE_KEY=MetaData:Source

REGEX=/var/log/data_01_\d+_(?P<date>\d+)\.LOG

[set_time]

INGEST_EVAL= _time = strptime(date,"%Y%m%d") + random() %1000

props.conf

[mysourcetype]

DATETIME_CONFIG =

SHOULD_LINEMERGE = false

LINE_BREAKER = ([\r\n]+)

NO_BINARY_CHECK = true

disabled = false

TRANSFORMS-time_set= get_date , set_time

However, it is timed in real time and the settings do not take effect.

The universal forwarder sends data to the indexer, and I put this setting in the indexer What's the problem?

[PickleRick]

First question - did you restart the indexer?

Second question - are you sure about the regex? The filename examples do not seem to match it.

Oh, and if you time your events at "date +/- random", you'll get your events scattered radomly around midnight that day (some before midnight, some after). You'll lose the order of events.

[noott211]

I think there's a typo because it's a mobile phone There is no problem with the regular expression, and turning the indexer off also does not solve it.

Props.conf

[source::/var/log/data*.LOG]

DATETIME_CONFIG =

SHOULD_LINEMERGE = false

LINE_BREAKER = ([\r\n]+)

NO_BINARY_CHECK = true

disabled = false

TRANSFORMS=get_date,set_time

transfroms.conf

[get_date]
SOURCE_KEY=MetaData:Source
REGEX=\/var\/log\/date_\d+_(?P<date>\d+)\.LOG

[set_time]

INGEST_EVAL= _time = strptime(date,"%Y%m%d") + random() %1000

And if there is a problem with the time range, is there any other way?

[PickleRick]

I've never tried putting TRANSFORMS alone as such into props.conf. It's always TRANSFORMS-something

So the relevant line to call your transforms should be something like

TRANSFORMS-datemanipulation=get_date,set_time

You can also try to do it in one go as shown in https://conf.splunk.com/files/2020/slides/PLA1154C.pdf (slide 29) so you don't get another indexed field that you don't need.

And I would drop this random() altogether.  Unfortunately, as you don't have any reliable time information you have to stick to just putting all your events in one timestamp.

Sorry, my initial remark about random() pushing some events to the day before was obviously wrong. You're only adding time to the midnight timestamp so the time will always be in the "next" day. But randomizing timestamp will mangle your order of events.

[noott211]

All methods were used but failed. The log time is still generated as the current time...

[PickleRick]

Check with btool if your resulting configuration looks as it should.

For quick debug you can add a transform adding some constant value to your event to check if it's being called at all.

Oh, and I see you have MetaData instead of Metadata (yes, it's case sensitive)

[noott211]

There is no problem when checking Btool Is it a matter of setting MetaData?