Getting Data In

Best way to extract _time from file name and text

NoSpaces
Contributor

Hello to everyone!
I have a curious situation:
I have log files that I collecting via SplunkUF
This log file does not contain a whole timestamp - one part of the timestamp is contained in the file name, and the other is placed directly in the event


As I found in the other answers, I have options.
1. INGEST_EVAL on the indexer layer: I did not understand how I could take one part from the source and glue it with _raw data
Link to the answer

2. Use handmade script to create a valid timestamp for events - this is more understandable for me, but it looks like "reinventing the wheel"

So the question is, may I use the first option if it is possible?
This is the an example of the source:

E:\logs\rmngr_*\24020514.log

* - some number
24 - Year
Month - 02
Day - 04
Hour - 14


And this is an example of the event:

45:50.152011-0,CONN,3,process=rmngr,p:processName=RegMngrCntxt,p:processName=ServerJobExecutorContext,OSThread=15348,t:clientID=64658,t:applicationName=ManagerProcess,t:computerName=hostname01,Txt=Clnt: DstUserName1: user@domain.com StartProtocol: 0 Success

45:50.152011 - Minute, Second and Subsecond

 

Labels (3)
0 Karma
1 Solution

isoutamo
SplunkTrust
SplunkTrust

Hi

you could look my old post https://community.splunk.com/t5/Getting-Data-In/How-to-apply-source-file-date-using-INGEST-as-Time/m....

You need to do small modifications to it. 

  1. Select correct format into 1st replace part to get year to hour part from source. 
  2. Replace tostring part with take your minutes to sub second from _raw e.g. substring/replace 
  3. modify format string to match your combined year to sub second format

You could test this like I have done on above post. If needed, don’t hesitate to ask more help.

Remember that INGEST_EVAL must be an one command only.

r. Ismo

View solution in original post

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

you could look my old post https://community.splunk.com/t5/Getting-Data-In/How-to-apply-source-file-date-using-INGEST-as-Time/m....

You need to do small modifications to it. 

  1. Select correct format into 1st replace part to get year to hour part from source. 
  2. Replace tostring part with take your minutes to sub second from _raw e.g. substring/replace 
  3. modify format string to match your combined year to sub second format

You could test this like I have done on above post. If needed, don’t hesitate to ask more help.

Remember that INGEST_EVAL must be an one command only.

r. Ismo

0 Karma

NoSpaces
Contributor

After a lot of tries, I finally did it
Looks simple when you know what to do
Thank you for advertising the substr function 😃

The final result is below



props.conf

[oce_file_rphost]
TRANSFORMS-oce_file_tc0 = oce_file_tc0
LINE_BREAKER = ()\d{2}:\d{2}.\d+-\d+,
SHOULD_LINEMERGE = false


transform.conf

[oce_file_tc0]
INGEST_EVAL = _time = strptime("20" + replace(source,".*\\\\(\d{8}).log","\1") + substr(_raw,0,12),"%Y%m%d%H%M:%S.%6Q")



Get Updates on the Splunk Community!

Stay Connected: Your Guide to June Tech Talks, Office Hours, and Webinars!

What are Community Office Hours?Community Office Hours is an interactive 60-minute Zoom series where ...

Splunk Observability Cloud’s AI Assistant in Action Series: Analyzing and ...

This is the second post in our Splunk Observability Cloud’s AI Assistant in Action series, in which we look at ...

Elevate Your Organization with Splunk’s Next Platform Evolution

 Thursday, July 10, 2025  |  11AM PDT / 2PM EDT Whether you're managing complex deployments or looking to ...