Strange text can come from a decoding mismatch, a valid Unicode character that looks unfamiliar, an invisible separator, or a font that lacks the expected glyph. Those causes need different fixes. Start by preserving the raw response, then inspect each layer from bytes to displayed characters instead of replacing suspicious symbols blindly.
First identify what “strange” means
A few common symptoms point in different directions:
| Symptom | Likely area to inspect |
|---|---|
| café instead of café | Bytes may have been decoded with the wrong character encoding. |
| A diamond with a question mark or replacement symbol | The decoder may have encountered invalid or unsupported byte sequences. |
| Two words that look identical but compare unequal | Unicode normalization or visually similar code points may differ. |
| Text is correct in one app but shows boxes in another | The font or rendering environment may lack a glyph. |
| A pasted value fails validation although it looks right | Hidden whitespace, direction marks, or nonbreaking spaces may be present. |
These are clues, not diagnoses. The same visible symptom can have several causes, so keep the original sample before trying to repair it.
Trace the text from source to screen
1. Preserve the original bytes and context
If the text arrived in an HTTP response, save the response body and inspect its headers. Record the declared media type and charset, if provided, and the client’s decoding behavior. If it came from a database, note the column encoding and the conversion path. A screenshot or copied snippet is not enough to reconstruct the original bytes.
Avoid repeatedly copying the text through editors that may normalize line endings or replace invalid sequences. Work on a copy and record which system produced it.
2. Check whether decoding happened once and with the intended encoding
Text files and network protocols need an agreement about how bytes map to characters. UTF-8 is common on the web, but legacy files and integrations can use other encodings. A mismatch can turn a valid byte sequence into mojibake. Re-encoding the already-corrupted visible string usually does not restore the original; go back to the original bytes if you can.
For web responses, verify the server’s Content-Type and the client’s parser. For a downloaded text file, check the application’s import option and source system’s export format. Do not assume that changing a browser declaration repairs data already decoded incorrectly upstream.
3. Inspect the actual Unicode code points
Characters can look alike while being different code points. A hyphen-minus (-), en dash (–), and em dash (—) have different characters and often different uses. Latin a and a visually similar character from another script may render nearly the same while failing a comparison. Combining accents can also be represented as a precomposed character or as a base letter followed by a combining mark.
A Unicode inspector can reveal a character’s code point and category. Use the Toolsnix Invisible Character Remover on non-sensitive text. Look for nonbreaking spaces, zero-width joiners, bidirectional marks, combining characters, and unexpected control characters. Their presence is not automatically malicious: they are legitimate in languages, typography, and emoji sequences. Interpret them in context.
4. Separate data correctness from font rendering
If the underlying string contains the expected code points but the screen shows a square or blank, the issue may be font coverage. Try a font that supports the script or symbol, and check the target operating system. Replacing the character with a visually similar symbol changes the data and may be wrong for search, names, code, or legal text.
If text appears in the wrong order, check for right-to-left scripts and bidirectional control behavior before removing marks. Arabic, Hebrew, and mixed-direction text can legitimately include directionality that affects display. Test with the intended rendering environment and preserve the original.
A small diagnostic example
Imagine an API should return the word café, but a log shows café. First compare the raw response bytes with the decoded string. If the bytes are UTF-8 for café but a client interprets them using a different encoding, the mismatch is in decoding. Fix the charset handling at the boundary, then confirm the stored and displayed value. Do not run a “repair” conversion on all data until you know whether the corruption happened once or is only a logging display issue.
Now consider a second symptom: a form rejects a copied identifier that looks like AB-123. Inspect code points. The visible separator might be a Unicode dash instead of ASCII hyphen-minus, or the value may contain a zero-width character. If the receiving system requires a specific format, normalize or validate according to that contract. Keep the original available for audit; do not silently alter names or natural-language content.
Normalize only when the data contract calls for it
Unicode normalization can make canonically equivalent text use a consistent representation. It does not make all visually similar characters equivalent, and it is not a universal security sanitizer. Choose a normalization form based on the application’s comparison and storage rules. For identifiers, define allowed characters and canonicalization explicitly. For prose, avoid destructive normalization that changes meaningful distinctions.
Likewise, trimming whitespace is appropriate for some user-entered identifiers but not for every text field. Leading spaces may be intentional in a code sample; line breaks may carry structure. Apply cleanup at a validated boundary and tell users what will change.
What not to do
- Do not replace every unfamiliar character with
?; that destroys information. - Do not assume any zero-width or directionality character indicates an attack.
- Do not declare that a file is UTF-8 based only on how it looks in one editor.
- Do not “fix” a database by converting every stored string without backups and a reproducible sample.
- Do not remove Unicode from names or non-English text simply because an older system mishandles it.
Inspect strings at each serialization boundary
Text often crosses several representations: bytes in a network response, a decoded string in application memory, a JSON escape sequence, a database value, and a rendered glyph. Identify where the unexpected character first appears. If the API response already contains an unexpected sequence, the browser font is unlikely to be the cause. If the raw value is correct but one view is wrong, inspect parsing and rendering instead.
JSON can escape a character as \uXXXX while representing the same Unicode code point as a literal character. Those forms may look different in source but decode to the same string. Do not compare raw JSON spelling when your actual requirement is equality of parsed text. Conversely, if you sign or hash the raw bytes, normalization or reserialization can change the byte sequence even when the displayed text looks identical. Use the canonicalization rules required by that protocol.
For CSV, check the file’s encoding and delimiter handling at import. A spreadsheet may open a UTF-8 file using a local default encoding, while a text editor correctly detects the bytes. Exporting again from the spreadsheet can preserve the already-corrupted display, so return to the original source when possible.
Keep a small diagnostic fixture
Create a test sample containing ordinary ASCII, an accented letter, a non-Latin script you support, an emoji sequence, a nonbreaking space, and punctuation. Store the expected code points or a trusted reference representation alongside it. Pass the sample through each boundary in the system and compare the result.
Do not use only an emoji as the test. Some emoji consist of multiple code points joined together; a system that counts visible glyphs may report a different number of code points. That difference can be correct. The expected behavior depends on whether your application counts user-perceived characters, code points, or UTF-16 code units.
When reporting a bug, include a minimized sample and the exact steps needed to reproduce it. Redact private names and tokens. State whether the character is wrong in the raw source, only in a log, or only in a rendered interface. This helps the next person investigate the right layer instead of applying a global cleanup rule.
Consider security without overreacting
Visually confusable characters can be relevant in usernames, identifiers, domain names, and source code. An attacker might use a lookalike to mislead a reader, but many Unicode differences are ordinary language and typography. Restricting every field to ASCII can exclude legitimate names and scripts. Define policies by field: a machine identifier may allow a narrow set, while a person’s display name should generally support the languages your service promises to handle.
If you detect bidirectional controls in a source-code or security-sensitive identifier, inspect the surrounding text and follow the relevant platform guidance. Preserve evidence before sanitizing. A safe display strategy can make controls visible to reviewers without corrupting the stored original.
Make a reproducible test across systems
When the problem only appears on one machine, compare the exact source and destination rather than relying on screenshots. Save a harmless test string in a file, compute or record its byte representation, and open it with the same applications involved. If one editor changes the value on save, compare before and after to find the first mutation.
For an API issue, add a minimal test response with the expected header and body. Test the server response independently of the browser UI, then inspect what the client library returns. If the server sends a JSON escape sequence, parse it and compare code points after parsing. This can distinguish a representation difference from actual data corruption.
For a database issue, query a safe sample through both the application and a database client. Check the connection encoding, column type, and import/export settings. Avoid “fixing” stored rows in place until you have a backup, a migration plan, and a query that identifies the affected records precisely.
Choose a correction at the source
If the server labels bytes incorrectly, correct the server’s content type or encoder. If an importer assumes the wrong character set, set the correct import encoding. If two strings use different canonical Unicode forms, normalize them according to the field’s comparison contract. If the glyph is absent, supply a suitable font or fallback.
A later layer can sometimes compensate for an earlier mistake, but that creates hidden complexity. For example, changing a frontend decoder may make one display look right while the database still stores corrupted text. Fix the earliest incorrect boundary you can verify and add a regression fixture so the problem does not return.
FAQs
What does � mean in a response?
It is the Unicode replacement character, often inserted when a decoder cannot represent or decode some input. Check the original bytes and decoding path; the replacement character alone cannot recover the missing value.
Can I convert mojibake back to the original text?
Sometimes, if the original bytes or a reversible sequence of incorrect decodings is known. If the data was replaced or discarded, the original characters may no longer be recoverable. Keep the source bytes whenever possible.
Why does text look right but fail an exact match?
The strings may contain different code points, normalization forms, whitespace, or invisible controls. Inspect the characters and compare against the application’s documented input rules.
Should I strip invisible Unicode characters?
Only when the field’s contract says they are invalid. Some invisible characters are required for correct script shaping, emoji, or bidirectional text. Validate by field type rather than using one global removal rule.
Next step
Capture the source, inspect response encoding, compare code points, and test rendering with a suitable font. Make the smallest correction at the layer where the mismatch begins, then preserve a before-and-after sample for regression checks.