topic Re: how to export csv with BOM ? in Getting Data In

how to export csv with BOM ?

newsm106 — Mon, 28 Sep 2020 16:13:11 GMT

Hi,

I have a csv file encoded UTF-8 from Scheduled Search. And MS Excel couldn't read the file Because without BOM I guess. How can I export csv file with BOM ??

I searched on Splunk and found this.
http://answers.splunk.com/answers/5610/can-the-export-function-export-csv-file-in-big5-charset

In the file
$SPLUNK_HOME/lib/python2.6/site-packages/splunk/appserver/mrsparkle/controllers/search.py
go to around line 266, where the
statement output = job.getFeed(asset,
http_method=jobFeedRequestMethod,
**kwargs) is.

Add the following code after that
line:
if 'isDownload' in kwargs:
    import codecs
    output = "".join((codecs.BOM_UTF8, output))
This should allow the files to be read
by Excel.

But My Python version is 2.7. So the code of Search.py is different.

How can I get CSV with BOM??

Regards,

Sam

Re: how to export csv with BOM ?

mcseem — Wed, 25 Feb 2015 14:17:19 GMT

Tried to append code after line 342 (output = job_lite.get(asset))
No result. Files still exporting without BOM

Re: how to export csv with BOM ?

SwatiApte — Tue, 13 Oct 2015 09:41:39 GMT

Hi,

When I added the above mentioned code in Python 2.7 search.py, and restarted splunk, the Web service did not start at all. The command prompt was stuck at "Waiting for the web service to be available". As soon as I removed the added three lines, it started working. Could you please let me know if there is any other way of doing the same, or whether the code is placed in the correct place.

Re: how to export csv with BOM ?

jeffland — Fri, 02 Sep 2016 10:35:54 GMT

For anyone interested in a solution for recent Splunk versions, go to the file

splunkhome/lib/python2.7/site-packages/splunk/rest/__init__.py

to line 648 where it says

def readall(self, blocksize=32768):
    """
    Returns a generator reading blocks of data from the response
    until all data has been read
    """
    response = self.response
    while 1:
        data = response.read(blocksize)
        if not data:
            break
        yield data

Change that to

def readall(self, blocksize=32768):
    """
    Returns a generator reading blocks of data from the response
    until all data has been read
    """
    response = self.response
    import codecs
    counter = 0;
    while 1:
        data = response.read(blocksize)
        if not data:
            break

        if counter == 0:
            data = "".join((codecs.BOM_UTF8, data))
            counter += 1

        yield data

and your exports come with BOM.

Re: how to export csv with BOM ?

ktc78 — Thu, 07 Nov 2019 01:34:08 GMT

jeffland's code works perfectly for previous splunk version

but in case of splunk v8.0.0,
I'm afraid python 3.7 is used

splunkhome/lib/python3.7/site-packages/splunk/rest/__init__.py

and raised error like below

Unrecoverable error in the server.
Traceback (most recent call last):
File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cpwsgi.py", line 184, in trap
return func(*args, **kwargs)
File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cpwsgi.py", line 277, in __next__
return next(self.iter_response)
File "/opt/splunk/lib/python3.7/site-packages/cherrypy/lib/encoding.py", line 99, in encoder
for chunk in body:
File "/opt/splunk/lib/python3.7/site-packages/splunk/rest/__init__.py", line 716, in readall
data = "".join((codecs.BOM_UTF8, data))
TypeError: sequence item 0: expected str instance, bytes found

Can I have code for python3.7?

Best regards,
Kang

Re: how to export csv with BOM ?

kichonei — Tue, 21 Apr 2020 01:39:41 GMT

hey, Kang

[python 2.7]
data = "".join((codecs.BOM_UTF8, data))

[python 3.7]
: Add 'b' in front.

data = b"".join((codecs.BOM_UTF8, data))