Why is Splunk lagging in latest logs?

[sc_admin11]

In my splunk cloud , when i search for index index="asterisk" and trigger any deployment to get logs for this index and select last 15 minute to get latest logs but no logs are there. But when i change time from last 15 minute to last 6 hours then i can see my latest logs, this is weird . Log time for asterisk is in UTC.
Can someone please help me here?

[sc_admin11]

@isoutamo  logs come on time in splunk but not showing in last 15 minutes. It was showing in last 6 hour logs. Looks like some time stamp issue 

[sc_admin11]

not getting your solution .
can you please tell me more?

[isoutamo]

I usually do that data onboarding process with separate splunk dev/test instance to set up props.conf & transfers.conf correctly.

• Install e..g splunk trial version to your workstation/some server (you could use also developer or dev/test license for longer period)
• Get some sample data from your onboarding system
• Use Splunk -> Settings -> Add Data 
  • Upload
  • Select file (your sample file)
  • Next
  • Update as need the next settings
    • Event Break
    • Timestamp
    • Advanced
  • When your events have handled properly remember save your new sourcetype definition with "Save As" button
  • Next -> Select host + index -> Review -> Submit
  • Then search events and do fixes and fine tunings if need as many times as need. You could remove events from your test index for meshing up things or use different host name to separate versions
• After you are happy with those events then just copy your props.conf, transforms.conf and create separate TA_xyz for those. Then install it to first full splunk instance counting from UF
• Time by time there could be some definitions on props.conf which need also install to UF too!
• Also you could test if there are some search time props/transforms and develop those also on this host/environment. Later on install those onto your production

I hope that this clarify what I'm meaning for onboarding?

r. Ismo

[isoutamo]

Then it's best to do onboarding process on your own test/dev instance and update props.conf to correct places after you have fixed that issue on your test splunk.

[isoutamo]

Hi

here is couple of answers/presentations/hints how to find the real issue and fix it

• https://lantern.splunk.com/Splunk_Platform/Product_Tips/Data_Management/Reducing_event_delay_in_Splu... contains also link to conf presentation
• https://conf.splunk.com/files/2019/slides/FN1570.pdf (another presentation)
• https://docs.splunk.com/Documentation/Splunk/latest/Troubleshooting/Troubleshootingeventsindexingdel...
• https://community.splunk.com/t5/Getting-Data-In/Why-is-my-Universal-Forwarder-showing-extreme-lag-or...

There are lot more those instructions how to find the reason behind that lag.

r. Ismo