topic Re: Logs are not getting indexed in Getting Data In

Logs are not getting indexed

gowthammahes — Mon, 20 May 2024 20:37:02 GMT

Hello Everyone,

Recently, I am trying to ingest the logs from my server. But it is not getting indexed. The log file which I am trying to ingest has different timestamp with same events.

Events in log file:

1712744099:{"jsonefd":"1.0","result":"1357","id":1} 1712744400:{"jsonefd":"1.0","result":"1357","id":1} 1712745680:{"jsonefd":"1.0","result":"1357","id":1} 1714518017:{"jsonefd":"1.0","result":"1378","id":1} 1715299221:{"jsonefd":"1.0","result":"1366","id":1}

I tried with crcsalt but still no luck. Kindly help if anyone faced this issue before.

I would like to ingest the events even the events are same with different timestamps.

Re: Logs are not getting indexed

richgalloway — Mon, 20 May 2024 20:05:41 GMT

Help us help you by providing more information.

How is the data being onboarded? IOW, what is the method for getting the events to Splunk?

Are there any errors in the logs?

How have you determined the events are not indexed?

Re: Logs are not getting indexed

inventsekar — Tue, 21 May 2024 01:36:34 GMT

Hi @gowthammahes

Are you trying to index this log file in indexer/search head directly

are you trying to read this file thru Universal Forwarder?

Re: Logs are not getting indexed

gowthammahes — Tue, 21 May 2024 08:19:10 GMT

We have installed the universal forwarder and the events are forwarded to intermediate forwarder from the splunk uf and then it sent to indexer.

But i could the host internal logs are being ingested into splunk. Only the file is not getting monitored

Re: Logs are not getting indexed

gowthammahes — Tue, 21 May 2024 08:22:24 GMT

How is the data being onboarded? IOW, what is the method for getting the events to Splunk?

-- Have installed the universal forwarder and added monitor stanza in it. and then uf will send the logs to intermediate fwd and then to indexer

Are there any errors in the logs?

There is no error even in debug mode

How have you determined the events are not indexed?

The index newly created and there is no events found in it. Have verified the log event timestamp and searched the events in search head at same time

Re: Logs are not getting indexed

richgalloway — Tue, 21 May 2024 12:21:24 GMT

Please share the inputs.conf and props.conf stanzas related to the input.

Have you searched the last chance index (usually 'main')? Have you searched all time, including the future, in case the timestamps are not interpreted correctly?

Re: Logs are not getting indexed

gowthammahes — Tue, 21 May 2024 18:40:35 GMT

inputs.conf:

[monitor:///var/log/json]
disabled = 0
index = app_prod
sourcetype = app-json
crcSalt = <SOURCE>

there is no props.conf

events:

As you said i searched with all time and no results found.

Re: Logs are not getting indexed

richgalloway — Tue, 21 May 2024 14:24:04 GMT

Missing props could be a problem. Try these settings.

[app:json] TIME_PREFIX = ^ TIME_FORMAT = %s LINE_BREAKER = ([\r\n]+) SHOULD_LINEMERGE = FALSE MAX_TIMESTAMP_LOOKAHEAD = 10 TRUNCATE = 10000 EVENT_BREAKER_ENABLE = TRUE EVENT_BREAKER = ([\r\n]+)

Note the change in sourcetype name. Avoid using hyphens in identifiers since they could be mistaken for the subtraction operator.

By default, Splunk will not search future times so it won't detect timestamps that were misinterpreted in that direction. Try index=app_prod earliest=-1y latest=+1y

Re: Logs are not getting indexed

gowthammahes — Tue, 28 May 2024 08:49:47 GMT

The issue has been resolved. Actually there was two tcp out indexer groups caused the issue. Adding _tcp_routing fixed the issue.