Logs are not getting indexed

[gowthammahes]

Hello Everyone,

Recently, I am trying to ingest the logs from my server. But it is not getting indexed. The log file which I am trying to ingest has different timestamp with same events.

Events in log file:

1712744099:{"jsonefd":"1.0","result":"1357","id":1}
1712744400:{"jsonefd":"1.0","result":"1357","id":1}
1712745680:{"jsonefd":"1.0","result":"1357","id":1}
1714518017:{"jsonefd":"1.0","result":"1378","id":1}
1715299221:{"jsonefd":"1.0","result":"1366","id":1}

I tried with crcsalt but still no luck. Kindly help if anyone faced this issue before. 

I would like to ingest the events even the events are same with different timestamps.

[gowthammahes]

The issue has been resolved. Actually there was two tcp out indexer groups caused the issue. Adding _tcp_routing fixed the issue.

[inventsekar]

Hi @gowthammahes 

Are you trying to index this log file in indexer/search head directly

OR

are you trying to read this file thru Universal Forwarder?

[gowthammahes]

We have installed the universal forwarder and the events are forwarded to intermediate forwarder from the splunk uf and then it sent to indexer.

But i could the host internal logs are being ingested into splunk. Only the file is not getting monitored

[richgalloway]

Help us help you by providing more information.

How is the data being onboarded?  IOW, what is the method for getting the events to Splunk?

Are there any errors in the logs?

How have you determined the events are not indexed?

[gowthammahes]

How is the data being onboarded?  IOW, what is the method for getting the events to Splunk?

-- Have installed the universal forwarder and added monitor stanza in it. and then uf will send the logs to intermediate fwd and then to indexer

Are there any errors in the logs?

There is no error even in debug mode

How have you determined the events are not indexed?

The index newly created and there is no events found in it.  Have verified the log event timestamp and searched the events in search head at same time

[richgalloway]

Please share the inputs.conf and props.conf stanzas related to the input.

Have you searched the last chance index (usually 'main')?  Have you searched all time, including the future, in case the timestamps are not interpreted correctly?

[gowthammahes]

inputs.conf:

[monitor:///var/log/json]
disabled = 0
index = app_prod
sourcetype = app-json
crcSalt = <SOURCE>

there is no props.conf

events:

1712744099:{"jsonefd":"1.0","result":"1357","id":1}
1712744400:{"jsonefd":"1.0","result":"1357","id":1}
1712745680:{"jsonefd":"1.0","result":"1357","id":1}
1714518017:{"jsonefd":"1.0","result":"1378","id":1}
1715299221:{"jsonefd":"1.0","result":"1366","id":1}

As you said i searched with all time and no results found.

[richgalloway]

Missing props could be a problem.  Try these settings.

[app:json]
TIME_PREFIX = ^
TIME_FORMAT = %s
LINE_BREAKER = ([\r\n]+)
SHOULD_LINEMERGE = FALSE
MAX_TIMESTAMP_LOOKAHEAD = 10
TRUNCATE = 10000
EVENT_BREAKER_ENABLE = TRUE
EVENT_BREAKER = ([\r\n]+)

Note the change in sourcetype name.  Avoid using hyphens in identifiers since they could be mistaken for the subtraction operator.

By default, Splunk will not search future times so it won't detect timestamps that were misinterpreted in that direction.  Try index=app_prod earliest=-1y latest=+1y