topic Re: How to set INGEST in Getting Data In

How to set INGEST

noott211 — Mon, 09 May 2022 03:53:21 GMT

There is no time field in my log and I tried to get time from the source file name

I tried the settings below

myfile /var/log/data_01_20220507

/var/log/data_02_20220506

transforms.conf

[get_date]

SOURCE_KEY=MetaData:Source

REGEX=/var/log/data_01_\d+_(?P<date>\d+)\.LOG

[set_time]

INGEST_EVAL= _time = strptime(date,"%Y%m%d") + random() %1000

props.conf

[mysourcetype]

DATETIME_CONFIG =

SHOULD_LINEMERGE = false

LINE_BREAKER = ([\r\n]+)

NO_BINARY_CHECK = true

disabled = false

TRANSFORMS-time_set= get_date , set_time

However, it is timed in real time and the settings do not take effect.

The universal forwarder sends data to the indexer, and I put this setting in the indexer What's the problem?

Re: How to set INGEST

PickleRick — Mon, 09 May 2022 06:49:17 GMT

First question - did you restart the indexer?

Second question - are you sure about the regex? The filename examples do not seem to match it.

Oh, and if you time your events at "date +/- random", you'll get your events scattered radomly around midnight that day (some before midnight, some after). You'll lose the order of events.

Re: How to set INGEST

noott211 — Mon, 09 May 2022 07:08:53 GMT

I think there's a typo because it's a mobile phone There is no problem with the regular expression, and turning the indexer off also does not solve it.

Props.conf

[source::/var/log/data*.LOG]

DATETIME_CONFIG =

SHOULD_LINEMERGE = false

LINE_BREAKER = ([\r\n]+)

NO_BINARY_CHECK = true

disabled = false

TRANSFORMS=get_date,set_time

transfroms.conf

[get_date]
SOURCE_KEY=MetaData:Source
REGEX=\/var\/log\/date_\d+_(?P<date>\d+)\.LOG

[set_time]

INGEST_EVAL= _time = strptime(date,"%Y%m%d") + random() %1000

And if there is a problem with the time range, is there any other way?

Re: How to set INGEST

PickleRick — Mon, 09 May 2022 07:19:18 GMT

I've never tried putting TRANSFORMS alone as such into props.conf. It's always TRANSFORMS-something

So the relevant line to call your transforms should be something like

TRANSFORMS-datemanipulation=get_date,set_time

You can also try to do it in one go as shown in https://conf.splunk.com/files/2020/slides/PLA1154C.pdf (slide 29) so you don't get another indexed field that you don't need.

And I would drop this random() altogether. Unfortunately, as you don't have any reliable time information you have to stick to just putting all your events in one timestamp.

Sorry, my initial remark about random() pushing some events to the day before was obviously wrong. You're only adding time to the midnight timestamp so the time will always be in the "next" day. But randomizing timestamp will mangle your order of events.

Re: How to set INGEST

noott211 — Mon, 09 May 2022 07:48:35 GMT

All methods were used but failed. The log time is still generated as the current time...

Re: How to set INGEST

PickleRick — Mon, 09 May 2022 08:06:25 GMT

Check with btool if your resulting configuration looks as it should.

For quick debug you can add a transform adding some constant value to your event to check if it's being called at all.

Oh, and I see you have MetaData instead of Metadata (yes, it's case sensitive)

Re: How to set INGEST

noott211 — Mon, 09 May 2022 09:10:19 GMT

There is no problem when checking Btool Is it a matter of setting MetaData?