Skip to content

Automatic charset detection #4

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Nyoroon wants to merge 4 commits into from
Closed

Automatic charset detection #4

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
3c9713a
Automatic charset detection
Nyoroon Oct 22, 2013
711e2e2
some fixes
Nyoroon Oct 22, 2013
a2f1fc4
forgot that chardet.detect() returns tuple
Nyoroon Oct 22, 2013
8399444
really it is dict
Nyoroon Oct 22, 2013
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,4 +51,5 @@ def read(filename):
irclogsearch = irclog2html.irclogsearch:main
""",
zip_safe=False,
install_requires=['chardet'],
)
10 changes: 5 additions & 5 deletions src/irclog2html/irclog2html.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
was written by Jeff Waugh and is available at www.perkypants.org
"""

# Copyright (c) 2005--2013, Marius Gedminas
# Copyright (c) 2005--2013, Marius Gedminas
# Copyright (c) 2000, Jeffrey W. Waugh

# Python port:
Expand Down Expand Up @@ -54,6 +54,7 @@
import shlex
import shutil
import sys
import chardet

try:
from urllib import quote
Expand Down Expand Up @@ -138,10 +139,9 @@ def decode(self, s):
if isinstance(s, unicode):
# Accept input that's already Unicode, for convenience
return s
try:
return s.decode('UTF-8')
except UnicodeError:
return s.decode('cp1252', 'replace')
else:
charset = chardet.detect(s)['encoding']
return s.decode(charset, 'replace')

def __iter__(self):
for line in self.infile:
Expand Down