Cleaning Up Full-Width, Half-Width and Invisible Characters

If Excel's VLOOKUP cannot find a value that is clearly there, a copied hashtag link does not work, or the Korean in a file name appears broken apart like ㅎㅏㄴㄱㅡㄹ, an invisible character difference is very likely the cause. The characters look the same on screen, but what the computer compares is the character code.

Characters you will often run into

• Full-width letters, digits and symbols: ＡＢＣ１２３！＠ in the Unicode range U+FF01–U+FF5E. They are typed in the full-width mode of Chinese, Japanese and Korean input methods and have different codes from half-width ASCII (U+0021–U+007E). Full-width digits may not be recognized as numbers in spreadsheets, and an email address containing a full-width ＠ is rejected as invalid.
• Full-width space (U+3000): a space as wide as a character, used in Chinese, Japanese and Korean documents.
• NBSP (U+00A0): the no-break space. HTML's   is this character, and it often gets mixed into text copied from web pages. Excel's TRIM function removes only ordinary spaces, so NBSP remains.
• Zero-width space (U+200B): it has zero width and is completely invisible. It creeps into text copied from web editors or messengers and breaks hashtags and URLs.
• BOM (U+FEFF): the byte order mark placed at the very start of a file. When you copy from a UTF-8 file with a BOM, for example one saved with Notepad, it hides in front of the first value of the first line and causes only that first value to fail comparisons.

Decomposed Korean (NFC and NFD)

Unicode allows 한 to be written either as a single code point (U+D55C), called NFC, or as three jamo letters ᄒ (U+1112) ᅡ (U+1161) ᆫ (U+11AB), called NFD. Both are displayed as the same character, but as strings they differ. File names created on macOS and some text brought over from macOS may contain NFD Korean, and in environments that cannot compose NFD for display, the letters appear broken apart. Even when it displays correctly, it is a different value from NFC Korean, so searching and deduplication fail. Accented Latin letters such as é can be stored in both forms in the same way.

The cleanup pipeline

1. Clean invisible characters and quotes — with the default options, NFC normalization recombines decomposed characters, zero-width characters and the BOM are removed, and special spaces such as NBSP become ordinary spaces. Curly quotes also become straight quotes.
2. Full-width ↔ half-width — with the direction set to 'Full-width → half-width', characters in U+FF01–U+FF5E are converted to ASCII. With 'Convert spaces too' on, full-width spaces also become ordinary spaces.
3. Trim leading and trailing spaces — removes the converted spaces that sit at the start or end of lines.

ＫＩＭ　minsu​@example.com
合計：１，２００円

The input above (a full-width space hides after ＫＩＭ and a zero-width space after minsu) is cleaned up as follows.

KIM minsu@example.com
合計:1,200円

Things to watch out for

• Full-width → half-width does not change Hangul, Chinese characters or kana, but it does convert full-width punctuation in Japanese and Chinese sentences (！ ？ ： ，) to half-width. For manuscripts whose appearance must be preserved, apply it only to the parts containing letters and digits.
• The won sign ￦ (U+FFE6) and the yen sign ￥ (U+FFE5) are outside the conversion range (U+FF01–U+FF5E) and are not changed.
• The 'Half-width katakana → full-width' option applies only when the direction is 'Half-width → full-width'.
• NFC normalization only recombines decomposed characters; it does not turn compatibility characters such as ① ㎏ Ⅳ into 1 kg IV. Such characters stay as they are.
• Conversely, 'Half-width → full-width' is used to fill in forms that require full-width input, such as Japanese forms. Ordinary spaces also become full-width spaces in that case, so turn off 'Convert spaces too' if you do not need that.
• Removing the zero-width joiner (U+200D) breaks combined emoji such as 👨‍👩‍👧 into separate emoji. For text where emoji must be kept, turn off 'Remove zero-width characters and BOM'.

Where do they come from?

• Japanese and Chinese websites, older Korean government documents and values typed in the full-width mode of an Asian input method easily contain full-width letters and digits.
• Copying a web page brings along NBSP written as  . It often sits at the end of sentences or between names.
• Text copied from link previews in messengers and social media, or from editors, sometimes contains zero-width spaces.

Excel's CLEAN function removes control characters with codes 0–31, so it does not handle NBSP, zero-width spaces or full-width characters. For values that cannot be fixed in the sheet, clean them up with this tool and paste them back.

How to find hidden characters

If you want to see which lines have a problem before cleaning up, set Filter lines to 'Keep only matching lines' with 'Regex' on, and enter [ ​　！-～] as the search text. Only lines containing those characters remain, so you can inspect the data that caused the problem. When you are done, switch this step off or delete it.

Checklist

• When a comparison fails, suspect invisible characters first.
• Put Clean invisible characters and quotes before the full-width conversion.
• Use full-width conversion only partially on Japanese and Chinese manuscripts.

Last updated: 2026-09-23