add_input_check fails to check input

[pdpinch]

from ZenDesk 91873 and ZenDesk 86186:

Pre-processing student submissions using add_input_check() hasn't worked since xqueuewatcher was upgraded to Python 3. It appears that the tokenizer is not working, probably due to an encoding issue.

xqueue-watcher/grader_support/gradelib.py

Lines 236 to 246 in a95cc96

	def _tokens(code):
	"""
	A wrapper around tokenize.generate_tokens.
	"""
	# Protect against pathological inputs: http://bugs.python.org/issue16152
	code = code.rstrip() + "\n"
	if isinstance(code, six.text_type):
	code = code.encode('utf8')
	code = "# coding: utf8\n" + code
	toks = tokenize.generate_tokens(six.BytesIO(code).readline)
	return toks

[pdpinch]

I'm hopeful PR #9 will fix this, but I notice that there aren't any tests at all for this feature.

[pdpinch]

This was fixed for Residential MITx in #11, but it needs tests before we can upstream it to edX.

[pdpinch]

In order to run this project locally with edX devstack and develop tests, it may help to review the documentation PR at https://github.com/edx/xqueue-watcher/pull/54

[pdpinch]

edX merged it anyway, in https://github.com/edx/xqueue-watcher/pull/58