Splunk is not indexing my text file

[Abilan1]

Hi Team,

It looks like splunk is not indexing my log file, If I create a test file then it is indexing it but not my log file.

input.conf

[monitor://D:\logs]
disabled = false
index = prd
sourcetype = sched

Log File sample:

2017-03-24 00:00:00,848 [Thread-2] WARN com.JobInitiator- executed CALL
2017-03-24 00:00:20,927 [Thread-2] WARN com.JobInitiator- executed CALL
2017-03-24 00:00:41,161 [Thread-2] WARN com.JobInitiator- executed CALL
2017-03-24 00:01:01,208 [Thread-2] WARN com.JobInitiator- executed CALL

Please help me on this issue. Thank You!

[danielsofoulis]

When searching select ALL TIME, this has tripped me up before especially if the test logs are old.

I'm not sure if you want to ingest the whole directory or a file named logs.

Try using creating a folder logs and putting your test file inside and then use the following stanza.

[monitor://D:\logs\*]

You could also confirm connectivity by using telnet
telnet SPLUNK_IP 9997

Another thing you could do is a one shot command. This will ingest the file regardless of whether the file has been ingested before.

SPLUNK_HOME\bin\splunk add oneshot D:\logs -index pad -sourcetype shed

http://docs.splunk.com/Documentation/Splunk/latest/Data/MonitorfilesanddirectoriesusingtheCLI

[jpass]

Check that the user splunk runs under has permission to access the folders & log files you want to index.

[Abilan1]

Hi ,

Permission looks good, that is not the issue.

[mattymo]

Hi Abilan1

You can try checking index=_internal tailreader with your log path in the GUI to see perhaps the tail reader is ignoring the file due to CRC checks or permissions otherwise, or check

./splunk list inputstatus to check for the files and the tailprocessor's status.

[mattymo]

your user is set to UTC in the Splunk GUI, correct? click the username dropdown (top right of your screen) > Account Settings > Set timezone

With my user set in UTC, I see these events at 3AM GMT in my Time field....is that what you are trying to achieve? Note the time in the event has not changed, just the Time field now has a local conversion in the gui.

can you please collect the output of ./splunk btool props list sched --debug on your forwarder and your indexer? ( not sure of your setup, I think you said forwarder)

you have the sched sourcetype updated in both a forwarder and indexer, correct?

`[splunker@n00bserver bin]$ ./splunk btool props list sched --debug
/home/splunker/splunk/etc/apps/search/local/props.conf [sched]

.........

/home/splunker/splunk/etc/apps/search/local/props.conf TIME_FORMAT = %Y-%m-%d %H:%M:%S,%f
/home/splunker/splunk/etc/apps/search/local/props.conf TIME_PREFIX = ^
/home/splunker/splunk/etc/system/default/props.conf TRANSFORMS =
/home/splunker/splunk/etc/system/default/props.conf TRUNCATE = 10000
/home/splunker/splunk/etc/apps/search/local/props.conf TZ = Brazil/East

........

`

Brasil event time, UTC Splunk web user, splunker sitting in Canada 😉

[Abilan1]

Hi ,

Thanks for your reply, I have verified the permission it looks fine. Also If I create any other test file in that folder that is getting indexed properly. so it doesn't seem to be a permission issue.

Checked the tail-processor's status, it says finished reading. No errors related to CRC.

Regards,
Abilan

[mattymo]

if the files are read, then maybe we are looking in the wrong spot?

does searching index=prd sourcetype=sched source=*\logs* searched over alltime find them?

[Abilan1]

Hi,

Hi,

Yes I see that logs are indexing, but If I want to search for current log then I have to search for last 4 hours. I mean indexing time stamp is 4 hour behind.

If I search for last 4 hours in the search, I can see the latest logs there.

We have the same forwarder on linux machines and that is giving proper time stamps. This is happening on windows server's universal forwarder. All our universal forwarder machines are in same time zone but not indexer.

Thanks ,
Abilan

[mattymo]

Sounds like you need to configure your sourcetype with timestamp settings. There are a few best practices when it comes to defining a sourcetype...but for now lets focus on the props.conf for timestamp recogonition.

What timezone should these logs be in?

can you share the props.conf you are using for sourcetype sched ?

./splunk btool props list sched --debug

Try adding TZ= <logTimezone> to the props to help Splunk determine the correct stamp.

[Abilan1]

Hi ,

My Indexer is in GMT. but all my forwarders are in Brazil time zone.

I didn't change anything in props.conf file after installation.

Also please let me know which timezone I need to add and any standard format for the same?

Thanks for your help!

Regards,
Abilan

[mattymo]

You need to set the timezone to that of the machines spitting the logs. IT best practice puts all machines in UTC, but alas, no one is perfect ;).

If i run date on the machine generating the logs, I want to match that timezone as that is the TZ it will stamp the logs with.

As long as Splunk knows the proper TZ at indextime, the GUI settings will allow the user to set their local timezone and the GUI will adjust for them. But you gotta get the initial TZ correct.

TZ =
* The algorithm for determining the time zone for a particular event is as
follows:
* If the event has a timezone in its raw text (for example, UTC, -08:00),
use that.
* If TZ is set to a valid timezone string, use that.
* If the event was forwarded, and the forwarder-indexer connection is using
the 6.0+ forwarding protocol, use the timezone provided by the forwarder.
* Otherwise, use the timezone of the system that is running splunkd.
* Defaults to empty.

http://docs.splunk.com/Documentation/Splunk/6.5.2/Admin/Propsconf

[Abilan1]

Hi ,

I have updated the below in my props.conf on forwarder machine.

[ sched ]
SHOULD_LINEMERGE=false
TIME_FORMAT=%Y-%m-%d %H:%M:%S,%f
TIME_PREFIX=^
TZ=UTC-3
MAX_TIMESTAMP_LOOKAHEAD=25

And restarted the splunk forwarder service, but still I don't see any change in time from Splunk web.

Regards,
Abilan

[mattymo]

try

TZ=America/Sao_Paulo

not sure that utc-3 value is legit...

[Abilan1]

Hi ,

I have updated this for UTC in my props.conf on forwarder machine.

[sched]
SHOULD_LINEMERGE=false
TIME_FORMAT=%Y-%m-%d %H:%M:%S,%f
TIME_PREFIX=^
TZ=Africa/Abidjan
MAX_TIMESTAMP_LOOKAHEAD=25

Still it is not working as expected. Logs are still coming in Brazil Time Zone.

Regards,
Abilan

[Abilan1]

Hi ,

Yes, I have tried with Brazil Time zone still it is not working!

[mattymo]

UTC is wrong. the logs are Brazil time. Set it to brazil, that way Splunk can translate it to your chosen timezone in the GUI

[mattymo]

Try this and set your timezone according to the machines that are spitting the sched logs to your windows box. Setting SHOULD_LINEMERGE, TIME_FORMAT, TIME_PREFIX, MAX_TIMESTAMP_LOOKAHEAD, TZ, and LINE_BREAKER in props.conf, should become part of all your sourcetyping. Splunk is really good at auto recognizing things..but you can gain indexing performance by not making it work so hard. Try using the add data wizard! it makes creating these configs real easy and lets you validate your choices.

[ sched ]
SHOULD_LINEMERGE=false
TIME_FORMAT=%Y-%m-%d %H:%M:%S,%f
TIME_PREFIX=^
TZ=UTC
MAX_TIMESTAMP_LOOKAHEAD=25