Oh god, umlauts!

Sweden has more lake data than most countries have lakes. However, they also have a 29 letter alphabet including ‘å’, ‘ä’ and ‘ö’. This results in lake names like Fräcksjön and Östra Helgtjärnen.

Goal
Combine two .csv files (data and metadata). The common ID is lake name.

Problem
Every lake name with a ‘swedish character’ would not match! Frustration!

Reason
Stack Overflow to the rescue.
Turns out in unicode, these special characters can be represented in two different forms.
1 character: ä (Normal-Form-Composed NFC) or
2 characters: a + double dots (Normal-Form-Decomposed NFD)
Therefore, Fräcksjön is 9 characters long in NFC, but 11 in NFD. Since grep in R matches characters and lengths, this leads to a problem.

My problem stemmed from the fact that one .csv was made on a PC (NFC) and one on MacOS (NCD).

Solution
You must switch the encoding on one of the files.

iconv(data, "UTF-8-mac", "UTF-8")

All that troubleshooting for one line of code. Here’s a great example:

Screen Shot 2015-11-11 at 5.04.24 PM

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s