Shift_JIS was created in 1982 as a collaboration between ASCII Corporation and Microsoft to encode Japanese characters on personal computers. It cleverly interleaves single-byte ASCII-compatible characters with double-byte JIS X 0208 characters, avoiding conflicts with existing control codes.

Microsoft adopted Shift_JIS for MS-DOS and Windows but gradually extended it with additional characters that customers demanded. This extended version was internally called “Code Page 932” (CP932), later standardized by IANA as Windows-31J. The problem: Microsoft never clearly distinguished their extensions from the original standard, and the name “Shift_JIS” became ambiguous.

In practice, when software claims to use “Shift_JIS,” it almost always means CP932. The pure JIS-standard Shift_JIS is rarely encountered in the wild. This creates a persistent source of confusion in encoding detection and conversion.

Shift_JIS and CP932 share the same fundamental structure: single-byte characters (0x00-0x7F and 0xA1-0xDF for half-width katakana) plus double-byte characters arranged in “rows” (ku) and “cells” (ten) from JIS X 0208. The difference lies in which rows are populated:

The core rows 1-8 and 16-84 are identical between the two encodings. The divergence occurs in the “gaps” — rows that JIS X 0208 left unassigned but Microsoft filled with vendor extensions. This is why most everyday Japanese text works identically in both encodings, but certain special characters fail when a strict Shift_JIS decoder encounters CP932-only characters.

Row 13 is the most notorious CP932 extension. NEC introduced these characters for their PC-9801 series in the 1980s, and Microsoft incorporated them into CP932. They include circled numbers, Roman numerals, unit symbols, and other frequently requested characters:

These characters are absent from standard Shift_JIS. A document containing ① (circled digit one, byte 0x8740) will decode perfectly in CP932 but produce an error or replacement character in a strict Shift_JIS decoder. This is the single most common cause of encoding issues when exchanging Japanese text between Windows and Unix/Mac systems.

The circled numbers are especially problematic because they are extensively used in Japanese business documents, legal texts, and everyday writing. Users are often shocked to learn that these “basic” characters are actually vendor extensions.

CP932 includes two sets of IBM-originated characters, and this creates a unique problem: some characters have two different byte sequences that map to the same Unicode code point.

The character 髙 (taka, the “tall” variant of 高) can be encoded as either the NEC-row byte sequence or the IBM-row byte sequence in CP932. Both decode to U+9AD9 in Unicode. However, when converting from Unicode back to CP932, the encoder must choose one representation. Microsoft chose to favor the NEC rows for round-trip compatibility, but other implementations may differ.

This duplication causes subtle bugs: byte-level string comparison may consider two CP932 strings “different” even though they contain identical text. Hash-based lookups, deduplication, and binary search can all break. The solution is to always normalize to Unicode for comparison, never compare raw CP932 bytes directly.

// The duplicate encoding problem:
// 髙 (U+9AD9) in CP932:
//   NEC row 89:  0xEEEF
//   IBM row 115: 0xFBFC
// Both are valid CP932, both map to the same Unicode character.
//
// Byte comparison: 0xEEEF ≠ 0xFBFC → "different"
// Unicode comparison: U+9AD9 === U+9AD9 → "same"
//
// Always convert to Unicode before comparing!

The WHATWG Encoding Standard, which governs how web browsers handle character encodings, made a pragmatic decision: the label “Shift_JIS” is treated as an alias for the Windows-31J (CP932) decoder. When a web page declares charset=Shift_JIS, browsers decode it using CP932 rules.

This means on the web, the distinction between Shift_JIS and CP932 is effectively erased. All six labels above trigger the same decoder. The WHATWG made this choice because virtually all “Shift_JIS” content on the web is actually CP932, and using a strict Shift_JIS decoder would break millions of pages containing NEC special characters.

However, this web-centric unification does not apply everywhere. Email (MIME), programming languages, and database systems may still distinguish between the two. Python's shift_jis codec is stricter than its cp932 codec. Java's Shift_JIS maps to JIS X 0208, while MS932 maps to CP932. These differences matter when processing data outside the browser.

# Python encoding behavior difference:
text = "①"  # Circled digit one (U+2460)

# CP932: works fine
text.encode('cp932')          # b'\x87\x40'

# Strict Shift_JIS: fails!
text.encode('shift_jis')      # UnicodeEncodeError

# Java equivalents:
# Charset.forName("Shift_JIS")  → JIS X 0208 based
# Charset.forName("MS932")      → CP932 / Windows-31J

Even when both Shift_JIS and CP932 contain the same character, they sometimes map to different Unicode code points. The most famous example is the wave dash problem:

These 7 mapping discrepancies (sometimes called the “wave dash problem” as a group) cause round-trip conversion failures. If you convert text from CP932 to Unicode and then to JIS-standard Shift_JIS (or vice versa), these characters will change identity. The wave dash (〜 vs ～) is the most visible and the most frequently encountered in practice — it appears in countless Japanese price ranges, dates, and expressions.

The root cause is that Microsoft and the JIS committee independently chose different Unicode code points for the same visual character. Neither choice is “wrong” — they simply reflect different mapping philosophies. Microsoft preferred fullwidth compatibility forms; JIS preferred semantically precise code points.