Getting Data In

CSV files with scientific notation fields

external_alien_
Explorer

I have a folder monitored by Splunk where CSV files are uploaded and sucked into Splunk. Splunk reads them no sweat and I can work with the data, the only problem is that the numerical values in the CSV files are all in Scientific notation and look for example like “2.7584000000000e+04” instead of simply “27584”. Splunk interprets them as numerical (not string) and I can fix this at search time with a few evals, but I have to do it for every search and was wondering if there's no way to fix this before the CSV files are indexed in Splunk? Say via editing props.conf?

Any help is much appreciated 😃

woodcock
Esteemed Legend

You can setup your eval statements as calculated fields using the EVAL- syntax here:

http://docs.splunk.com/Documentation/Splunk/6.4.2/Admin/Propsconf

0 Karma

external_alien_
Explorer

But doesn't that mean I have to know all the field names beforehand? Seeing as I have a large number of fields with values in scientific notation this is unfeasible, not to mention that field names may vary 😃
Is it theoretically possible to identify all values that contain say "e+" and rework them as plain decimal?

0 Karma

woodcock
Esteemed Legend

In that case, you need to create a macro using foreach and then use it whenever you need it. That is the best that you can do. Unfortunately, you cannot make the macro call automatic.

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Mastering Threat Intelligence in ES 8.5, Splunk AI Assistant v2, and More from Splunk ...

Splunk Lantern is Splunk’s customer success center that provides practical guidance from Splunk experts on key ...

Break the Build: Inside the KubeDoom Lounge at .conf26

    You step up to the machine. The pixelated corridors of a certain 1993 FPS load in front of you, EMP Pulse ...

Splunk Auto Ingestion Parallel Pipeline Scaling

Why this feature matters Many Splunk environments experience ingestion pressure long before the host is fully ...