Getting Data In

How to set INGEST

noott211
Path Finder

 

There is no time field in my log and I tried to get time from the source file name

I tried the settings below

myfile /var/log/data_01_20220507

/var/log/data_02_20220506

.

.

 

transforms.conf

[get_date]

SOURCE_KEY=MetaData:Source

REGEX=/var/log/data_01_\d+_(?P<date>\d+)\.LOG

[set_time]

INGEST_EVAL= _time = strptime(date,"%Y%m%d") + random() %1000

 

props.conf

[mysourcetype]

DATETIME_CONFIG =

SHOULD_LINEMERGE = false

LINE_BREAKER = ([\r\n]+)

NO_BINARY_CHECK = true

disabled = false

TRANSFORMS-time_set= get_date , set_time

 

However, it is timed in real time and the settings do not take effect.

The universal forwarder sends data to the indexer, and I put this setting in the indexer What's the problem?

 

 

 

0 Karma

PickleRick
SplunkTrust
SplunkTrust

First question - did you restart the indexer?

Second question - are you sure about the regex? The filename examples do not seem to match it.

Oh, and if you time your events at "date +/- random", you'll get your events scattered radomly around midnight that day (some before midnight, some after). You'll lose the order of events.

 

0 Karma

noott211
Path Finder

 

I think there's a typo because it's a mobile phone There is no problem with the regular expression, and turning the indexer off also does not solve it.

 

 


Props.conf

[source::/var/log/data*.LOG]

DATETIME_CONFIG =

SHOULD_LINEMERGE = false

LINE_BREAKER = ([\r\n]+)

NO_BINARY_CHECK = true

disabled = false

TRANSFORMS=get_date,set_time

 

 

transfroms.conf

 

[get_date]
SOURCE_KEY=MetaData:Source
REGEX=\/var\/log\/date_\d+_(?P<date>\d+)\.LOG

 

[set_time]

INGEST_EVAL= _time = strptime(date,"%Y%m%d") + random() %1000

 

And if there is a problem with the time range, is there any other way?

 

0 Karma

PickleRick
SplunkTrust
SplunkTrust

I've never tried putting TRANSFORMS alone as such into props.conf. It's always TRANSFORMS-something

So the relevant line to call your transforms should be something like

TRANSFORMS-datemanipulation=get_date,set_time

You can also try to do it in one go as shown in https://conf.splunk.com/files/2020/slides/PLA1154C.pdf (slide 29) so you don't get another indexed field that you don't need.

And I would drop this random() altogether.  Unfortunately, as you don't have any reliable time information you have to stick to just putting all your events in one timestamp.

Sorry, my initial remark about random() pushing some events to the day before was obviously wrong. You're only adding time to the midnight timestamp so the time will always be in the "next" day. But randomizing timestamp will mangle your order of events.

0 Karma

noott211
Path Finder

All methods were used but failed. The log time is still generated as the current time...

0 Karma

PickleRick
SplunkTrust
SplunkTrust

Check with btool if your resulting configuration looks as it should.

For quick debug you can add a transform adding some constant value to your event to check if it's being called at all.

Oh, and I see you have MetaData instead of Metadata (yes, it's case sensitive)

0 Karma

noott211
Path Finder

There is no problem when checking Btool Is it a matter of setting MetaData?

0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud’s AI Assistant in Action Series: Analyzing and ...

This is the second post in our Splunk Observability Cloud’s AI Assistant in Action series, in which we look at ...

Elevate Your Organization with Splunk’s Next Platform Evolution

 Thursday, July 10, 2025  |  11AM PDT / 2PM EDT Whether you're managing complex deployments or looking to ...

Splunk Answers Content Calendar, June Edition

Get ready for this week’s post dedicated to Splunk Dashboards! We're celebrating the power of community by ...