Reporting

How to map inputcsv on files over 10000 lines?

nick405060
Motivator

Given a specified n number of csvs, I want to input and append them:

| makeresults | eval count=mvrange(0,n,1) | mvexpand count | eval filename=strftime(relative_time(now(),"-"+tostring(count)+"mon"),"directory\\%Y-%m.csv")
| map maxsearches=n search="| inputcsv $filename$"

But this does not work. It caps each csv file at 10000 lines whereas outside of mapping, inputcsv will load the full csv. Any suggestions? My file sizes are under 3MB so it is kind of crazy this does not work in Splunk.

mapping:

search="|inputcsv start=0 file.csv | append [|inputcsv start=10000 file.csv]" 

It does not fix the problem either, It still only brings in the first 10k lines for each file.

1 Solution

nick405060
Motivator

Solved. Change maxout under [subsearch] in etc/sytem/local/limits.conf. Not sure what the long-term negative ramifications of this are.

View solution in original post

nick405060
Motivator

Solved. Change maxout under [subsearch] in etc/sytem/local/limits.conf. Not sure what the long-term negative ramifications of this are.

nick405060
Motivator

I also ran into this problem where this subsearch was capping results at 10000 behind the scenes:

| makeresults | map maxsearches=999999 search="search index=\"accesscontrol\" earliest=1555800000 latest=1556400000" | ....

and adjusting maxout fixed it.

(Lol I had the same problem again, Googled it, found this SA page, and was like tyvm whoever this helpful poster is!! Then realized it was me... )

0 Karma

nick405060
Motivator

Problem stated in another way:

| makeresults | map maxsearches=1 search="| inputcsv \"file.csv\"" | search <ITEM AT LINE 9999>

brings back results, but

| makeresults | map maxsearches=1 search="| inputcsv \"file.csv\"" | search <ITEM AT LINE 10001>

does not.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...