Splunk Search

Lookup csv with Ñ/ü/ä/í... characters (German/Spanish/French)

marina_rovira
Contributor

Hello all,

I have some csv files that I'm updating to splunk as lookup files, but there are some german/spanish/french characters that are not being recognized. Recently I found out the text iso necessary for this and I've changed the sourcetype for the index.

How can I do it for the lookups? If not possible, maybe it's easier to change them to index data?

Thank you in advance,

0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi marina_rovira,
If your lookups are built from indexed data, you could rebuild them using a search with outputlookup command.
otherwise the easiest way is to export them (or to take the original csv files), modify them in Excel or Notepad++ and then upload the updated files.

Bye.
Giuseppe

View solution in original post

ddrillic
Ultra Champion

The following seems to be a good approach -

-- I used a specialized converter to change it to UTF-8 and now it works fine.

It's at Lookup files with foreign characters

alt text

0 Karma

jraso
Explorer

Hi Marina,

What it has worked for me is to edit the file with Notepad++
In the Coding menu, select "Convert to UTF-8 and save as a new file.CSV.
Open the new file.CSV, select "Code as ANSI" and save again.
With this new file.CSV, inputlookup show all characters OK.

0 Karma

jraso
Explorer

Update: I've tried again with Notepad++ "Convert to UTF-8" and save.
It worked also perfecty.

marina_rovira
Contributor

One thing here, if in one index, I have Informàtica and in the lookup I have Infromatica because of the conversion, Will I have problems in the search and mix them?

0 Karma

ddrillic
Ultra Champion

Not sure Marina about the Informàtica's data - we need to find out in which character encoding your data is...

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi marina_rovira,
If your lookups are built from indexed data, you could rebuild them using a search with outputlookup command.
otherwise the easiest way is to export them (or to take the original csv files), modify them in Excel or Notepad++ and then upload the updated files.

Bye.
Giuseppe

marina_rovira
Contributor

Hey, they are not build from indexed data. I just export and move them directly to the lookup directory in splunk app search. This is because they should be static, they are updated with minimal changed once a month, and for looking info with them easier, but there is no previous data indexed.

What I need is something as the ISO field in the sourcetype data for the lookup tables.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi marina_rovira,
instead to move lookup files, try to newly import them in your SHs , eventually using Lookup Editor App, so you can immediately see result.
Bye.
Giuseppe

0 Karma

marina_rovira
Contributor

I've started that, but I have a little question. The good thing about moving it directly is that I have a script doing it, which implies no manual work from my part.

Adding it as new by the app, then I have to update it manually every month? Or it will recognize if the script overwrites the file?

Thank you for your help.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi marina_rovira,
what's the output of your script?
if it's something that Splunk can recognize, you can ingest every month the script's results in Splunk (with the correct management of characters) and then update your lookup using outputlookup command.
Bye.
Giuseppe

0 Karma

marina_rovira
Contributor

Do you have near you some understanding information about how to sue the OUTPUT thing?

The output of my script is a csv file directly.

0 Karma

gcusello
SplunkTrust
SplunkTrust

You could ingest in Splunk the csv files in an index and then create a search with outputlookup command that one time a month rebuild your lookup

index=this_new_index earliest=-24h latest=now
| fields field1 field2 ... fieldn
| outputlookup yourlookup.csv

scheduled after your script execution.

Bye.
Giuseppe

marina_rovira
Contributor

Sorry for bother you one more time.

I've done everything and it seems it works, just one more question. Now the csv file will be injected as an index, where or how can I schedule this rebuilt search?

Thank you in advance.

0 Karma

marina_rovira
Contributor

I think I found it 🙂 Tahnk you anyway for all the help.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi Marina,
If this answer satisfies your question, please accept it.
Thank you.
Giuseppe

0 Karma

marina_rovira
Contributor

Hello! Sorry for bringing this live again, but I have one question.

I seemed to work, but now, I have again the bad characters for the languages special ones. The problem is I don't remember how to modify it. Also when I try to open a lookup, it says me he file is too big to open it.

Can you help me please?

Thank you

0 Karma

marina_rovira
Contributor

I think I'm starting to understand what you meant.

I will try this. Thank you 🙂

0 Karma

marina_rovira
Contributor

Just, one more question.

This file, for example, I can not pick the time range for exporting, as is a view itself. So, every month, if there is any change, is because there's something new, something to add.
If I use an index, I will have the information repeated every time right?
Do you know some way in bash or in splunk itself, to not have all the lines repeated? (besides the dedup command in the search)

Thank you so much, you're helping me a lot.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi marina_rovira,
If you ingest csv in an index one time a month and use for it the automatic Splunk Timestamp, you'll have all the csv raw with the same timestamp in index.
So choosing last month as time period you're sure to take only the last one raw for each one and you can replace the full lookup.
If instead you want update only the different from lookup raws, you could run a search like this:

index=your_index earliest=-mon latest now NOT [ | inputlookup your_lookup.csv ] 
| table field1 field2 ... fieldn
| outputlookup your_lookup.csv append=true

Bye.
Giuseppe

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...