topic Re: How to export "Structured Logs" from Splunk to CSV file in Getting Data In

How to export "Structured Logs" from Splunk to CSV file

psychogyiokosta — Thu, 08 Aug 2019 14:58:46 GMT

Hello.

I am new with Splunk, I have the following question/issue:

My goal is to parse a raw log file with Splunk and save and download/extract the new generated structured log file as a csv file.

I ran this SPL search:

host="vm-nc-23" index="deeplog" sourcetype="hdfs_log" Content=*
| eval rex_template=replace(Content,"[_]*[-]*\d+","*")
| cluster t=0.5 labelonly=true labelfield=Template match=termlist field=rex_template
| stats count AS Occurences, values(rex_template) AS REGEX_Expressions by Template

Which outputs a set of n(=35) templates (the 'Content' field of logs, where variables/parameters are replaced with wildcards).

The templates can be viewed at the "Statistics" Tab and can be exported as a CSV file.

Instead, I would like to export the structured logs which include the additional field 'Template' among others (Time, PID, Component, Content, Level, etc.) from our SPL search. The structured logs can be vieweb in "Events" Tab.

I am confident that there isn't an option for exporting this from Splunk GUI. Any advice on how I could achieve this?

Thank you in advance!

Re: How to export "Structured Logs" from Splunk to CSV file

psychogyiokosta — Thu, 08 Aug 2019 15:10:42 GMT

This is an image of what i can export (templates): https://paste.pics/6BWYU

This is an image of what i want to export (structured logs): https://paste.pics/6BX0W

Re: How to export "Structured Logs" from Splunk to CSV file

nareshinsvu — Fri, 09 Aug 2019 04:38:35 GMT

Do you mean that you can only see the lognames in your events tab? But you want to see the contents of the log files from splunk search commands?

You have to index those logs so that you can extract what you wanted and export into csvs. You can't export log-files as such from splunk.

Re: How to export "Structured Logs" from Splunk to CSV file

psychogyiokosta — Fri, 09 Aug 2019 15:31:34 GMT

I would like to export the structured log lines shown in Events Tab (https://paste.pics/6BWYU).

Instead, i can only download the templates, shown in Statistics (https://paste.pics/6BX0W)

Re: How to export "Structured Logs" from Splunk to CSV file

nareshinsvu — Mon, 12 Aug 2019 01:58:21 GMT

Not sure if you are looking to outputlookup into csv?

<your base search> | outputlookup structured_logs.csv

After this, you can read your lookupfile using

|inputlookup outputlookup structured_logs.csv

or find it under

%SPLUNK_HOME%\etc\apps\search\lookups folder

Re: How to export "Structured Logs" from Splunk to CSV file

psychogyiokosta — Mon, 19 Aug 2019 17:44:37 GMT

Using the following SPL query:

host="vm-nc-23" index="deeplog" sourcetype="hdfs_log" Content=*

| eval rex_template=replace(Content,"[_]*[-]*\d+","*")

i found out that i am able to extract manually the .csv file of the "structured" logs, that is the logs with the fields that Splunk defines.
In these fields, there is also 1 field called "rex_template which extracts the Template of a given log line, as we have defined it here:

eval rex_template=replace(Content,"[_]*[-]*\d+","*")

This way of selecting the templates looks very basic. What would we do if we want to extract the structured logs, where the rex_template column is better structured?

an example of the current "basic" template extraction would be this:

Received block blk* src: /...😘 dest: /...😘 of size * Receiving block blk* src: /...😘 dest: /...😘

This is "ugly", instead i would like to have for example:

Received block blk<> Receiving block blk<>

(Note: src: /...😘 dest: /...😘 of size * we would like not to have this as well, it is not useful for my purpose)

Thank you all in advance.

Re: How to export "Structured Logs" from Splunk to CSV file

psychogyiokosta — Thu, 29 Aug 2019 08:47:37 GMT

That's helpful to know thank you. Still the lookup file i export as you described are the "templates", that is the lines generated in Statistics tab.

Is there a way to export the structured logs from the Events tab, similarly to the way you described above?

Also, since Splunk interface (i use a web browser to use Splunk, i do not have it installed locally) shows in the Event tab the 'structured logs' this means that Splunk saves the logs somewhere. However if i access my VM where Splunk is installed, i try:

grep -r 'string i want to find' in the Splunk's filepath and i get no results with files containing this string, indicating that there is no file with the structured logs to download.

How can i get access to the Event tabs content?

Thank you