What's a good way to detect file encoding in python that's not chardet?
-
What's a good way to detect file encoding in python that's not chardet?
Really it's just to identify latin1 or similar, so perhaps I should just replace it with some specialized logic for that.
-
What's a good way to detect file encoding in python that's not chardet?
Really it's just to identify latin1 or similar, so perhaps I should just replace it with some specialized logic for that.
@marcusxms
Any byte sequence is valid latin1, so there's no reliable way to detect it. You can only guess, maybe with the help of dictionaries or at least tables of characters that commonly appear within words.Depending on your usecase it may be enough to test for ASCII and UTF-8 first, and assume latin1 if both fail.
-
@marcusxms
Any byte sequence is valid latin1, so there's no reliable way to detect it. You can only guess, maybe with the help of dictionaries or at least tables of characters that commonly appear within words.Depending on your usecase it may be enough to test for ASCII and UTF-8 first, and assume latin1 if both fail.
@inguin Alright, thanks for the tip!
-
What's a good way to detect file encoding in python that's not chardet?
Really it's just to identify latin1 or similar, so perhaps I should just replace it with some specialized logic for that.
It seems one could use charset-normalizer at least in some cases.
