Dashboards & Visualizations

How do I deal with a log that contains HTML

JShood
Explorer

We have a log that contains an HTML formatted dump of memory variables at the time an error occurred.

Is there any way to have splunk show the HTML formatted log data in search results? I'm planning on parsing out a few of the basics like error code, location, user etc as standard splunk variables, but the detail will need to remain in HTML.

If HTML can't be used, how do other people deal with large amounts of debug data? Can splunk better process XML for instance?

Tags (2)
1 Solution

Michael_Wilde
Splunk Employee
Splunk Employee

Any reason why your output has to be HTML? I have used a command line browser, such as lynx or elinks wired up to a scripted input in Splunk that polls a webpage and retrns the results...something like . I actually didn't want all the HTML tags in my data. Why do you need them in yours?

If you look at http server status, in my case I wanted that big table of http objects and all the stats. With lynx I rendered it, with awk, I formatted each line with a timestamp and key value pairs. Might be an approach for you if you have control over how the log comes out.

View solution in original post

gkanapathy
Splunk Employee
Splunk Employee

As Michael said, it would be better if you didn't have it in HTML.

Nevertheless, Splunk can take and index the HTML just fine, and should be able to do field extractions on the data as well. (Your regexes will be ugly to get around the HTML.) The HTML will not be rendered in the Splunk UI, but I suppose that should be unnecessary, since what you do with Splunk is extract the data, then analyze and display it from there.

gkanapathy
Splunk Employee
Splunk Employee

That's probably better. Splunk is really for collecting data out of text files, then analyzing and processing the data, and the UI is made for that, not for rendering documents. In principle, a whole new UI could be built to do whatever with the indexed data, but that's not really the expected use of the standard web UI.

0 Karma

JShood
Explorer

Thanks for the suggestions. I think I may be trying to use splunk for something it's not intended. For each error in our web app we do a detailed dump of all variables, stack traces and sql states at the time of the error. This is rendered in a 500k+ HTML page. My intention was to add some header info that could be easily parsed by splunk and then from the splunk UI be able to view the nicely formatted HTML dump created by our application. I'm now considering splitting the log into two files, an overview for splunk to process and then a separate detailed dump. Any other suggestions?

0 Karma

Michael_Wilde
Splunk Employee
Splunk Employee

Any reason why your output has to be HTML? I have used a command line browser, such as lynx or elinks wired up to a scripted input in Splunk that polls a webpage and retrns the results...something like . I actually didn't want all the HTML tags in my data. Why do you need them in yours?

If you look at http server status, in my case I wanted that big table of http objects and all the stats. With lynx I rendered it, with awk, I formatted each line with a timestamp and key value pairs. Might be an approach for you if you have control over how the log comes out.

Get Updates on the Splunk Community!

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...

Observability Highlights | January 2023 Newsletter

 January 2023New Product Releases Splunk Network Explorer for Infrastructure MonitoringSplunk unveils Network ...