Getting Data In

How can I parse 2 sets of CSVs in one file?

hylam
Contributor

How could I parse this?

section1String
field1,field2,field3
value1,value2,value3
value1,value2,value3
value1,value2,value3
section2String
field4,field5,field6,field7
value4,value5,value6,value7
value4,value5,value6,value7
value4,value5,value6,value7

The number of value lines is unknown. Please handle 4 cases
a) The header lines are known strings
b) The header lines satisfy a regex
c) The header lines are 1 line below a known string
d) The header lines are 1 line below a string satisfying a regex

0 Karma
1 Solution

woodcock
Esteemed Legend

Like this:

| inputcsv TwoInOne.csv | rex "(?<filebreaker>SomRegExHere)" | streamstats current=t count(filebreaker) AS fileID | where fileID=0 outputcsv FirstHalf.csv
| inputcsv TwoInOne.csv | rex "(?<filebreaker>SomRegExHere)" | streamstats current=t count(filebreaker) AS fileID | where fileID=1 outputcsv SecondHalf.csv

| inputcsv FirstHalf.csv | append [|inputcsv SecondHalf.csv]

Note that this approach should work for any number of concatenated files, provided that filebreaker is the same.

View solution in original post

0 Karma

woodcock
Esteemed Legend

Like this:

| inputcsv TwoInOne.csv | rex "(?<filebreaker>SomRegExHere)" | streamstats current=t count(filebreaker) AS fileID | where fileID=0 outputcsv FirstHalf.csv
| inputcsv TwoInOne.csv | rex "(?<filebreaker>SomRegExHere)" | streamstats current=t count(filebreaker) AS fileID | where fileID=1 outputcsv SecondHalf.csv

| inputcsv FirstHalf.csv | append [|inputcsv SecondHalf.csv]

Note that this approach should work for any number of concatenated files, provided that filebreaker is the same.

0 Karma

woodcock
Esteemed Legend

Like this:

| inputcsv TwoInOne.csv | head 4 | outputcsv FirstHalf.csv
| inputcsv TwoInOne.csv | tail 4 | outputcsv SecondHalf.csv

| inputcsv FirstHalf.csv | append [|inputcsv SecondHalf.csv]

hylam
Contributor

your suggestion should work on fixed number of lines. i have edited the question. thx

0 Karma

woodcock
Esteemed Legend

The best thing to do is split the file on the outside of splunk.

0 Karma

hylam
Contributor

Are there any line numbers in splunk? Can a splunk search grep for "field[1-3]" and "field[4-7]" and expand a subsearch to "4"?

0 Karma

woodcock
Esteemed Legend

There is an internal field called _serial which is a line-number. I will think more about this approach if my other answer does not work (let me know with followup comments).

0 Karma

hylam
Contributor

plz see edit1

0 Karma
Get Updates on the Splunk Community!

ATTENTION!! We’re MOVING (not really)

Hey, all! In an effort to keep this Slack workspace secure and also to make our new members' experience easy, ...

Splunk Admins: Build a Smarter Stack with These Must-See .conf25 Sessions

  Whether you're running a complex Splunk deployment or just getting your bearings as a new admin, .conf25 ...

AppDynamics Summer Webinars

This summer, our mighty AppDynamics team is cooking up some delicious content on YouTube Live to satiate your ...