topic Re: How to create Table from log with csv format? in Dashboards & Visualizations

.

svenrieder — Wed, 23 Feb 2022 17:05:56 GMT

This Question got deleted.

Re: How to create Table from log with csv format?

gcusello — Mon, 07 Feb 2022 11:49:57 GMT

you have to index your csv using the INDEXED_EXTRACTION mode, for more infos see at https://docs.splunk.com/Documentation/Splunk/8.2.4/Data/Extractfieldsfromfileswithstructureddata

in few words, you have to put in your props.conf:

[your_sourcetype] HEADER_FIELD_LINE_NUMBER=1 TIMESTAMP_FIELDS=TimeStamp FIELD_DELIMITER=, FIELD_QUOTE="

I don't see in your sample the TimeStamp, if you haven't don't use the TIMESTAMP_FIELDS option.

The easiest approach is to try to index a sample of your csv files using the "Add Data" function [Settings -- Add Data], that guides you in the sourcetype definition.

Ciao.

Giuseppe

Re: How to create Table from log with csv format?

svenrieder — Mon, 07 Feb 2022 12:41:49 GMT

Hi @gcusello,

I can't do that, as I work in a large Company and dont have the rights to do that.

Isn't it possible to do that with a Splunk Search? If not it would also be possible for me to write every row into a seperate log-file if its possible that way.

Re: How to create Table from log with csv format?

gcusello — Mon, 07 Feb 2022 12:53:31 GMT

Hi @svenrieder,

yes, you could extract all the fields at search time

index=your_index | rex "\"(?<Plugin_Name>[^\"]+)\",\"(?<Severity>[^\"]+)\",\"(?<IP_Address>]\"]+)\",\"(?<NetBIOS_Name>[^\"]+)\",\"(?<DNS_Name>[^\"]+)\",\"(?<MAC_Address>[^\"]+)\",\"(?<Repository>[^\"]+)\",\"(?<Agent_ID>[\"]+)\"" | table PluginName Severity IP_Address NetBIOS_Name DNS_Name MAC_Address Repository Agent_ID

Ciao.

Giuseppe

Re: How to create Table from log with csv format?

svenrieder — Mon, 07 Feb 2022 18:40:44 GMT

Hi @gcusello

Thats perfect thank you! Now the string with that Information is in the Field "Data" From the Event:

How do you tell the rex where the information to filter is?

Re: How to create Table from log with csv format?

gcusello — Tue, 08 Feb 2022 07:17:18 GMT

Hi @svenrieder,

as you can see at https://docs.splunk.com/Documentation/Splunk/8.2.4/SearchReference/Rex

you have to little modify the search:

index=your_index | rex field=Data "\"(?<Plugin_Name>[^\"]+)\",\"(?<Severity>[^\"]+)\",\"(?<IP_Address>]\"]+)\",\"(?<NetBIOS_Name>[^\"]+)\",\"(?<DNS_Name>[^\"]+)\",\"(?<MAC_Address>[^\"]+)\",\"(?<Repository>[^\"]+)\",\"(?<Agent_ID>[\"]+)\"" | table PluginName Severity IP_Address NetBIOS_Name DNS_Name MAC_Address Repository Agent_ID

addinf "field=Data" to the rex command.

Ciao.

Giuseppe

Re: How to create Table from log with csv format?

PickleRick — Tue, 08 Feb 2022 07:23:30 GMT

Ask "the splunk guys" to define proper extractions. After all that's the point of using splunk - to have actionable data. If they're just ingesting everything "as is" without caring what it's used for, you're kind of wasting your license.

And you can define your own field extractions so you don't have to type in the rex command every time.

Re: How to create Table from log with csv format?

svenrieder — Wed, 09 Feb 2022 09:49:05 GMT

Hi @gcusello

Somehow it is still not working. It is not showing the results in the table:

Re: How to create Table from log with csv format?

gcusello — Wed, 09 Feb 2022 10:03:06 GMT

Hi @svenrieder,

this means that somehow there's some difference in the log format, you have to identify these differences and modify the regex used for the extractions.

If you could identify the events with different logs and share them I could help you in regex update.

Ciao.

Giuseppe

Re: How to create Table from log with csv format?

svenrieder — Wed, 09 Feb 2022 11:02:15 GMT

Hi @gcusello

Thanks for your Help! The Event looks like this:

Re: How to create Table from log with csv format?

gcusello — Wed, 09 Feb 2022 11:09:39 GMT

Hi @svenrieder,

I found that sometimes some of the fields could be empty.

This can be managed using "*" instead "+" in the regex, something like this:

Ciao.

Giuseppe

Re: How to create Table from log with csv format?

svenrieder — Wed, 09 Feb 2022 14:04:16 GMT

Hi @gcusello

Okay it worked, thanks! I now used different attributes for testing the Search:
Name, Website, Country

The log is read by Splunk without making a new line which means the csv that looked like this:
"Thomas","thomas.com","England"
"Luca","luca.com","USA"

Looks like this in Splunk:
"Thomas","thomas.com","England" "Luca","luca.com","USA"

So the Search is showing the Table with the first line only:

How do I have to configure the log input or the splunk search so it lists every line?

Re: How to create Table from log with csv format?

gcusello — Wed, 09 Feb 2022 14:06:50 GMT

Hi @svenrieder,

let me understand: in each event you have the header (that contains all the field names) and the values or only on some events?

my regex is done for events with only the values, without the field names.

if you have some events with only the values and others with header and values, I hint to skeak with the people who created the parser because there's an error in parsing!

Anyway, in this case you could use two different regexes: one for events with only values (the one I already sent you) and another one for the events with both field names and values:

index=your_index | rex field=Data "\"(?<Plugin_Name>[^\"]*)\",\"(?<Severity>[^\"]*)\",\"(?<IP_Address>]\"]*)\",\"(?<NetBIOS_Name>[^\"]*)\",\"(?<DNS_Name>[^\"]*)\",\"(?<MAC_Address>[^\"]*)\",\"(?<Repository>[^\"]*)\",\"(?<Agent_ID>[\"]*)\"" | rex field=Data "^(\"[^\"]*\",){7}\"[^\"]*\"\s+\"(?<Plugin_Name>[^\"]*)\",\"(?<Severity>[^\"]*)\",\"(?<IP_Address>[^\"]*)\",\"(?<Netbios_Name>[^\"]*)\",\"(?<DNS_Name>[^\"]*)\",\"(?<MAC_Address>[^\"]*)\",\"(?<Repository>[^\"]*)\",\"(?<Agent_ID>[^\"]*)\"" | table PluginName Severity IP_Address NetBIOS_Name DNS_Name MAC_Address Repository Agent_ID

Ciao.

Giuseppe

Re: How to create Table from log with csv format?

svenrieder — Wed, 09 Feb 2022 14:45:19 GMT

Hi @gcusello

I can delete the top line out before I send it to Splunk, as the Search is easier to understand and looks nicer.
So in the Field Data I only have the Values without the Header.

My Question is, this
| rex field=Data "\"(?<Name>[^\"]*)\",\"(?<Website>[^\"]*)\",\"(?<Country>[^\"]*)\""
Only reads out the first line.

The thing is, that when I write the values from the csv-File into a Log-File, it adds all rows together so if it was:

Line 1
Line 2
Line 3

In the Log it will be:
Line 1 Line 2 Line 3

So if I have this in the CSV
"Thomas","thomas.com","England"
"Luca","luca.com","USA"
In the Log-File and therefore also in the Field of the event it looks like this:
"Thomas","thomas.com","England" "Luca","luca.com","USA"

Now my Question was, if it is possible to make a table with only one Event, which has all Lines in it and if yes, how do the Lines have to be Formatted in the Log and how does the rex look like to show it like this:

(Here I made 2 Events, for every line a sepearate Event)