topic Re: How to index HTML files? in Getting Data In

How to index HTML files?

sergeyvinnik — Wed, 25 Jun 2014 18:55:13 GMT

I have source log files with HTML formatting. After indexing I get 12000 lines in one record.
I need to remove HTML mark-ups to index every particular line in the log file.
Can I do it in Splunk?
What source type should I use?

Re: How to index HTML files?

chanfoli — Wed, 25 Jun 2014 21:20:51 GMT

If I understand correctly, you want to strip HTML tags from an input file. I do not believe that any built-in sourcetype or extraction is going to handle this. I would approach this by pre-processing the logs. Many scripting languages have facilities to strip tags out of streams/files. It is probably something you could do with a SEDCMD or regex transform, but that may not be the best way to go.

Regards,
Sean

Re: How to index HTML files?

sergeyvinnik — Wed, 25 Jun 2014 22:59:43 GMT

Can I use TRANSFORM with following REGEX?

/<[a-zA-Z_/=]*>/ /g

It should replace all tags like .. by spaces

Re: How to index HTML files?

edschembor — Thu, 26 Jun 2014 18:48:13 GMT

Remember tho that this transformation will have to occur everytime you run the query. It would be much more efficient to create script to pre-process and just do it once

Re: How to index HTML files?

LukeMurphey — Thu, 10 Jul 2014 06:06:48 GMT

There is an app called Website Input that was designed to pull information from websites. That might handle your case if the HTML files are accessible via an HTTP server.