topic Re: File can't be indexed in Getting Data In

File can't be indexed

serviceinfrastr — Thu, 08 Nov 2018 15:38:28 GMT

Hi Team,

I have on file (is the picture) that are unable to catch and index

i have this configuration in my input.conf

[monitor://D:\eo\contLive\logs\job*.log]
sourcetype = progress:inter
index = progress
crcSalt = <SOURCE>
disabled = false

[monitor://D:\eo\contLive\logs\*.log]
sourcetype = progress:contlive
index = progress
disabled = false

the source type progress:inter have been created in a specific TA (bellow the props.conf)

[ progress:inter ]
SHOULD_LINEMERGE=false
NO_BINARY_CHECK=true
CHARSET=UTF-8
REPORT-intervention-status=REPORT-intervention-status
category=Structured
disabled=false
TIME_FORMAT=%d/%m/%Y %H:%M:%S.%3N

i already try to do only this input and the specific file (jobstatus.log) is not indexed

   [monitor://D:\eo\contLive\logs\*.log]
    sourcetype = progress:contlive
    index = progress
    disabled = false

Many thanks for your help

Re: File can't be indexed

Rob2520 — Thu, 08 Nov 2018 16:44:03 GMT

did you check the permissions on the file? This stanza [monitor://D:\eo\contLive\logs\*.log] should catch all files ending with .log. If you want job*.log with a different sourcetype try this inputs.conf
[monitor://D:\eo\contLive\logs\*.log]
 sourcetype = progress:contlive
 index = progress
 disabled = false
blacklist = job[\d\w]+\.log$

[monitor://D:\eo\contLive\logs\job*.log]
 sourcetype = progress:inter
 index = progress
 crcSalt = <SOURCE>
 disabled = false

Re: File can't be indexed

serviceinfrastr — Fri, 09 Nov 2018 07:37:07 GMT

Thanks @Rob2520 for your reply.

I have the same problem with the blacklist setting.

For the permission it's the same than others files (windows server, all the folder files permissions are herited)

All also try to copy this file in job2.log to check if the problem was from the quick usage of the file by my application but the job2.log was also not indexing.

😕

Re: File can't be indexed

inventsekar — Fri, 09 Nov 2018 07:51:06 GMT

from this host, do you receive other logs fine? was the UF newly installed?
when the UF last communicated with the DS?
when you did the app push from DS to this UF, was it successfull?!?!

Re: File can't be indexed

serviceinfrastr — Fri, 09 Nov 2018 08:03:23 GMT

Hi @iventsekar

Y receive fine other log :

The UF was in 6.6.4 i just update to the last one v7.2.0 but the problem is the same

The last connection was "Few seconds" 🙂
all the modification done to the DS (input.conf) was sucessful pushed to the UF

In the splunkd.log on the UF log i can see that my config was googd and the file was found.

11-09-2018 08:43:31.448 +0100 INFO  WatchedFile - Will begin reading at offset=5934250 for file='D:\eo\contLive\logs\jobStatus.log'.

but not indexed ...

Re: File can't be indexed

inventsekar — Fri, 09 Nov 2018 09:10:09 GMT

by looking at your props.conf, that jobStatus.log is looks like a simple/normal file.
maybe, try this.. simply remove the props.conf and see if the file gets ingested.
then, write the props.conf file line by line(after understanding each line's meaning)

we also had a similar issue. we did this above method and it worked fine.

Re: File can't be indexed

teunlaan — Fri, 09 Nov 2018 12:20:12 GMT

Are you sure it did't get indexed? It it has an offset it did read it earlier.
Could there be something wrong with the timestamp in you logfile? (try searching way back "all Time" , but also in the future "now > +20y"

If there are timestamp issues, it also could have been deleted imidiatly if it is oudsite your accepted timerange

Re: File can't be indexed

serviceinfrastr — Fri, 09 Nov 2018 13:01:16 GMT

Hi @teunlaan
Your right !!!

My event was timestamped in september

But the correct date is today 09 November, French and Us time missmatch

are there any solution to correct this ?

Re: File can't be indexed

teunlaan — Fri, 09 Nov 2018 15:12:24 GMT

hmmm , your time_format in looks ok .

Have you tried it to insert it with the GUI (add data) , to see if it recognizes the timestamp correctly?
I guess the problem is the " | " that is connected with the time

(@ this moment I don't have access to a machine too do some test, sorry)

Re: File can't be indexed

FrankVl — Fri, 09 Nov 2018 15:31:52 GMT

Sounds like your TIME_FORMAT setting is not being applied (as that setting does seem to have the correct format). Instead, Splunk takes a guess, and mixes up days and months.

Where have you deployed the props.conf? If you ingest using a UF, the props needs to be on your indexer, to apply that TIME_FORMAT setting.

Re: File can't be indexed

serviceinfrastr — Mon, 12 Nov 2018 15:08:48 GMT

Hi,

All was correct with

[ progress:inter ]
SHOULD_LINEMERGE=true
disabled=false
TZ=Europe/Paris
TIME_FORMAT=%d/%m/%Y %H:%M:%S.%3N

Re: File can't be indexed

serviceinfrastr — Mon, 10 Dec 2018 12:55:42 GMT

Hi All,

I have a strange issue.

Since the 01 december 2018 the date format is not recognize.

My source in input.conf

[monitor://D:\eo\contLive\logs\job*.log]
sourcetype = progress:inter
index = progress
crcSalt = <SOURCE>
disabled = false

The TA of my progress:inter in the props.conf

[ progress:inter ]
SHOULD_LINEMERGE=true
disabled=false
TZ=Europe/Paris
TIME_FORMAT=%d/%m/%Y %H:%M:%S.%3N

And the

the date Month and day is not the good one.

10 December is 12 October

Many thanks