Getting Data In

How can I parse 2 sets of CSVs in one file?

hylam
Contributor

How could I parse this?

section1String
field1,field2,field3
value1,value2,value3
value1,value2,value3
value1,value2,value3
section2String
field4,field5,field6,field7
value4,value5,value6,value7
value4,value5,value6,value7
value4,value5,value6,value7

The number of value lines is unknown. Please handle 4 cases
a) The header lines are known strings
b) The header lines satisfy a regex
c) The header lines are 1 line below a known string
d) The header lines are 1 line below a string satisfying a regex

0 Karma
1 Solution

woodcock
Esteemed Legend

Like this:

| inputcsv TwoInOne.csv | rex "(?<filebreaker>SomRegExHere)" | streamstats current=t count(filebreaker) AS fileID | where fileID=0 outputcsv FirstHalf.csv
| inputcsv TwoInOne.csv | rex "(?<filebreaker>SomRegExHere)" | streamstats current=t count(filebreaker) AS fileID | where fileID=1 outputcsv SecondHalf.csv

| inputcsv FirstHalf.csv | append [|inputcsv SecondHalf.csv]

Note that this approach should work for any number of concatenated files, provided that filebreaker is the same.

View solution in original post

0 Karma

woodcock
Esteemed Legend

Like this:

| inputcsv TwoInOne.csv | rex "(?<filebreaker>SomRegExHere)" | streamstats current=t count(filebreaker) AS fileID | where fileID=0 outputcsv FirstHalf.csv
| inputcsv TwoInOne.csv | rex "(?<filebreaker>SomRegExHere)" | streamstats current=t count(filebreaker) AS fileID | where fileID=1 outputcsv SecondHalf.csv

| inputcsv FirstHalf.csv | append [|inputcsv SecondHalf.csv]

Note that this approach should work for any number of concatenated files, provided that filebreaker is the same.

0 Karma

woodcock
Esteemed Legend

Like this:

| inputcsv TwoInOne.csv | head 4 | outputcsv FirstHalf.csv
| inputcsv TwoInOne.csv | tail 4 | outputcsv SecondHalf.csv

| inputcsv FirstHalf.csv | append [|inputcsv SecondHalf.csv]

hylam
Contributor

your suggestion should work on fixed number of lines. i have edited the question. thx

0 Karma

woodcock
Esteemed Legend

The best thing to do is split the file on the outside of splunk.

0 Karma

hylam
Contributor

Are there any line numbers in splunk? Can a splunk search grep for "field[1-3]" and "field[4-7]" and expand a subsearch to "4"?

0 Karma

woodcock
Esteemed Legend

There is an internal field called _serial which is a line-number. I will think more about this approach if my other answer does not work (let me know with followup comments).

0 Karma

hylam
Contributor

plz see edit1

0 Karma
Get Updates on the Splunk Community!

New in Splunk Observability Cloud: Automated Archiving for Unused Metrics

Automated Archival is a new capability within Metrics Management; which is a robust usage & cost optimization ...

Calling All Security Pros: Ready to Race Through Boston?

Hey Splunkers, .conf25 is heading to Boston and we’re kicking things off with something bold, competitive, and ...

What's New in Splunk Observability - July 2025

What’s New?  We are excited to announce the latest enhancements to Splunk Observability Cloud as well as what ...