diff --git a/docs/how-to/builtins.rst b/docs/how-to/builtins.rst index 1403a4e35..a517340c6 100644 --- a/docs/how-to/builtins.rst +++ b/docs/how-to/builtins.rst @@ -1,104 +1,136 @@ Implementing Python Built-ins in JavaScript =========================================== +Python's builtins give Python its trademark magical feel. If you're new to Python, read up on ``_ + +Most builtins have already been added to the project, but many are do not quite match the original +implementation exactly. Some may not handle certain types of inputs correctly. In addition, new builtins +may arrive with the latest and greatest Python version. This guide should serve as your field manual for +adding, updating, and navigating our implementations. + +Process +------- + +The first thing to do when adding anything to Batavia is to play around a bit with it in the Python REPL. +Here's an example using ``list()``:: + + >> list() + [] + >> list((1, 2, 3, 4)) + [1, 2, 3, 4] + >> list(4) + Traceback (most recent call last): + File "", line 1, in + TypeError: 'int' object is not iterable + +Your goal is to find out how the function responds to various inputs and outputs. You may also +want to consult the offical documentation. Once you're a little familiar, you can start to add your +implementation to Batavia. + General Structure ------------------ +***************** JavaScript versions of Python built-in functions can be found inside the ``batavia/builtins`` -directory in the Batavia code. Each built-in is placed inside its own file. +directory in the Batavia code. Each built-in is placed inside its own file. These builtins are +designed to be used only inside Batavia, as such they need to ensure they are being used in +a compatible manner. -.. code-block:: javascript +Each builtin function will receive arguments and keyword arguments and needs to handle them, +even if the result is throwing an error. Args should be an array, and kwargs should be a +JavaScript object. The first thing to do is check that both were passed in. - // Example: a function that accepts exactly one argument, and no keyword arguments +Let's take a look at an example using the ``list()`` builtin + +.. code-block:: javascript - var = function(, ) { - // These builtins are designed to be used only inside Batavia, as such they need to ensure - // they are being used in a compatible manner. + // List accepts exactly one argument and no keyword arguments - // Batavia will only ever pass two arguments, args and kwargs. If more or fewer arguments - // are passed, then Batavia is being used in an incompatible way. - // See: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Functions/arguments + var list = function(args, kwargs) { + // Always add this code. if (arguments.length !== 2) { throw new builtins.BataviaError.$pyclass("Batavia calling convention not used."); } - // We are now checking if a kwargs object is passed. If it isn't kwargs will be null. Like - // obj.keys() in Python we can use Object.keys(obj) to get the keys of an object. If the - // function doesn't need support any kwargs we throw an error. - if (kwargs && Object.keys(kwargs).length > 0) { - throw new builtins.TypeError.$pyclass("() doesn't accept keyword arguments."); - } +This code ensures that the function can handle keyword arguments. Next, we need to validate the arguments are +correct. We can use JavaScript's ``Object.keys()`` to get the keys of an object. If we can't accept certain +args or kwargs, we will check the Python REPL to see what kind of error should be thrown and throw it. - // Now we can check if the function has the supported number of arguments. In this case a - // single required argument. - if (!args || args.length !== 1) { - throw new builtins.TypeError.$pyclass("() expected exactly 1 argument (" + args.length + " given)"); - } +.. tabs:: - // If the function only works with a specific object type, add a test - var obj = args[0]; - if (!types.isinstance(obj, types.)) { - throw new builtins.TypeError.$pyclass( - "() expects a (" + type_name(obj) + " given)"); - } + .. group-tab:: Python REPL - // actual code goes here - Javascript.Function.Stuff(); - } + .. code-block:: + + >> list(a=1) + TypeError: list() doesn't accept keyword arguments. + >> list(1, 2, 3) + TypeError: list() expected exactly 1 argument (3 given) - .__doc__ = 'docstring from Python 3.4 goes here, for documentation' + .. group-tab:: Batavia Code - modules.export = + .. code-block:: javascript + if (kwargs && Object.keys(kwargs).length > 0) { + throw new exceptions.TypeError.$pyclass("() doesn't accept keyword arguments."); + } -Adding Tests ------------- + if (!args || args.length !== 1) { + throw new exceptions.TypeError.$pyclass("() expected exactly 1 argument (" + args.length + " given)"); + } -The tests corresponding to Batavia implementations of built-ins are available inside -``tests/builtins``. The Batavia test infrastructure includes a system to check the compatibility of -JavaScript implementation of Python with the reference CPython implementation. + // If the function only works with a specific object type, add a test + var obj = args[0]; + if (!types.isinstance(obj, types.)) { + throw new exceptions.TypeError.$pyclass( + "() expects a (" + type_name(obj) + " given)"); + } -It does this by running a test in the Python interpreter, and then running the same code using -Batavia in the Node.js JavaScript interpreter. It will compare the output in both cases to see if -they match. Furthermore the test suite will automatically test the builtin against values of all -data types to check if it gets the same response across both implementations. + Useful functions are ``types.isinstance``, which checks for a match against a Batavia type or list, + of Batavia types, ``types.isbataviainstance``, which checks for a match against any Batavia instance, + ``Object.keys(kwargs)`` for dealing with kwargs, and JavaScript's ``for in``, ``for of``, and + ``Array.forEach`` loops for iterating over the JavaScript arrays and objects. -In many cases these tests will not cover everything, so you can add your own. For an example look at -the ``test_bool.py`` file in ``tests/builtins``. You will see two classes with test cases, -``BoolTests`` and ``BuiltinBoolFunctionTests``. Both derive from ``TranspileTestCase`` which -handles running your code in both interpreters and comparing outputs. + Note also the format for errors: ``throw new exceptions..$pyclass``. -Let's look at some test code that checks if a the Batavia implementation of ``bool`` can handle a -bool-like class that implements ``__bool__``. +Returning a value +***************** -.. code-block:: Python +Builtins implement Python functions and should return a Python object. +Batavia implementations of all Python types are located in ``/batavia/types.js``. +JavaScript imports use the ``require`` keyword and can be imported inline or at +the top of the file. Inline imports can be preferable in some cases. - def test_bool_like(self): - self.assertCodeExecution(""" - class BoolLike: - def __init__(self, val): - self.val = val +.. code-block:: javascript - def __bool__(self): - return self.val == 1 - print(bool(BoolLike(0))) - print(bool(BoolLike(1))) - """) + ... -The ``assertCodeExecution`` method will run the code provided to it in both implementations. This -code needs to generate some output so that the output can be compared, hence the need to print the -values. + Tuple = require('../types.js').Tuple + return new Tuple(my, results, here) + } +Documentation +************* -Process ----------- +Finally, add the docstring to the function object. In JavaScript, like in Python, functions +are first-class objects and can have additional properties. + +.. code-block:: javascript -For a given function, run `functionname.__doc__` in the Python 3.4 repl + list.__doc__ = 'docstring from Python 3.x goes here, for documentation' -Copy the docstring into the doc + module.exports = list -Run the function in Python 3.4 +Tests +***** -Take a guess at the implementation structure based on the other functions. +No implemenation for a project like this is complete without tests. Check out the other sections for +more details on test structure. Tests are located in ``/tests`` in a similar folder structure to the +core code, and most test files have already been created. Some things that should almost always be +tested: -Copy the style of the other implemented functions +* Write a test or three to ensure your function returns the correct output with some normal inputs. +* Think of a few weird inputs that could throw off your code (or future code). Test them. +* If you are throwing an error (excluding ``BataviaError``) anywhere, write a test that tries to throw it. +* If you accounted for an edge case (look for an ``if`` statement), test it. +* Check out the `official documentation `_ for more edge cases. \ No newline at end of file diff --git a/docs/how-to/contribute-docs.rst b/docs/how-to/contribute-docs.rst index 3ef7f5d32..ff8bc02d1 100644 --- a/docs/how-to/contribute-docs.rst +++ b/docs/how-to/contribute-docs.rst @@ -28,6 +28,6 @@ Create the static files: :: $ make html -Check for any errors and,if possible, fix them. +Check for any errors and, if possible, fix them. The output of the file should be in the ``_build/html`` folder. Open the file you changed in the browser. diff --git a/docs/how-to/contribute-tests.rst b/docs/how-to/contribute-tests.rst new file mode 100644 index 000000000..fc40ec302 --- /dev/null +++ b/docs/how-to/contribute-tests.rst @@ -0,0 +1,123 @@ +Implementing Tests in Batavia +============================= + +Basic Test Structure +-------------------- + +Batavia's job is to run a browser-compatible Python compiler, which takes valid Python as input and runs it. +Therefore, tests should test that the output of the Batavia compiler matches the output of CPython:: + + print('Hello') # Code to test + Hello # CPython output + Hello # Batavia output + # Outputs match. Test passes! + +This test structure is simple and effective. It's used in almost every test we've written. + +Adding Tests +------------ + +In many cases, existing tests will not cover everything. Feel free to add your own! + +The tests corresponding to Batavia implementations of built-ins are available inside +``tests/builtins``. The Batavia test infrastructure includes a system to check the compatibility of +JavaScript implementation of Python with the reference CPython implementation. + +These tests all derive from ``TranspileTestCase``, which handles running your code in both interpreters +and comparing outputs. For an example, look at the ``test_bool.py`` file in ``tests/builtins``. You +will see two classes with test cases, ``BoolTests`` and ``BuiltinBoolFunctionTests``. Both derive +from ``TranspileTestCase``. + +Let's look at some test code that checks if a the Batavia implementation of ``bool`` can handle a +bool-like class that implements ``__bool__``. + +.. code-block:: Python + + def test_bool_like(self): + self.assertCodeExecution(""" + class BoolLike: + def __init__(self, val): + self.val = val + + def __bool__(self): + return self.val == 1 + print(bool(BoolLike(0))) + print(bool(BoolLike(1))) + """) + +The ``assertCodeExecution`` method will run the code provided to it in both implementations. This +code needs to generate some output so that the output can be compared, hence the need to print the +values. **Code that is not being printed is not being tested.** + +Finally, ``print()`` is an imperfect creature for tests. Some things in Python aren't guaranteed to +print out in the same order every time, like sets dictionaries. Tests should be structured to compensate, +for instance by converting to a sorted list. See also the output cleaners section below. + +Template +-------- + +.. code-block:: python + + def test__(self): + # Valid Python code to be tested. + code = """ + print('>>> print()') + print() + """ + self.assertCodeExecution(code) + +This code block provides a printout of the code being run as well as the output of the code, +which can be very useful for debugging in test cases where more than a few lines of code are being run. + +Testing for Errors +------------------ + +Since we're testing the compiler, we need to ensure that errors for all of the builtins are thrown correctly. +We also want to ensure that we're not getting the wrong errors in our tests. Simply include a try/except +block in your test. + +.. code-block:: python + + def test_some_error(self): + code = """ + try: + code_that_raises_a_ValueError() + except ValueError as err: + print(err) + print("Test complete!") + """ + self.assertCodeExecution(code) + +Remember to catch the specific error you want, and then print the error. Then, print a success message to +validate that your except block didn't crash as well. **Code that is not being printed is not being tested.** + +Output Cleaners +--------------- + +In some cases, the test output will vary. ``TranspileTestCase`` will automatically apply some common output +cleanup for you. Some cases will need more or less cleanup. If you run your Python code directly in the REPL, +and the output differs from the test case output, you may need to modify what cleanup steps are being run. + +As such, ``assertCodeExecution`` accepts optional ``js_cleaner`` and ``py_cleaner`` objects. These can be provided by +the ``@transform`` decorator, located in ``tests/utils/output_cleaners.py``. Here's an example: + +.. code-block:: python + + @transform(float_exp=False) + def test_some_floats(self, js_cleaner, py_cleaner): # + Cleaner objects as arguments + code = ... + self.assertCodeExecution(code, js_cleaner=js_cleaner, py_cleaner=py_cleaner) # + Cleaner objects again + +This code means that the output of floating-point numbers will not be normalized using a regex. Refer to other +test cases and the docstring for ``@transform`` for more examples. + +Node/Python Crashes +------------------- + +If the CPython or JavaScript code crashes outright, UnitTest struggles. For instance, +``confused END_FINALLY`` in the middle of your test output tends to mean that the JavaScript code threw an +uncaught exception, causing Node to stop. It's hard for UnitTest to pull the details out of this type of thing +since that error occurred in Node, not Python. + +These types of errors will often appear above the test case as a crash report instead of in the usual section for the +output of your test's print() statements. Look there for clues. \ No newline at end of file diff --git a/docs/how-to/index.rst b/docs/how-to/index.rst index 1453651ac..86f80bd5f 100644 --- a/docs/how-to/index.rst +++ b/docs/how-to/index.rst @@ -12,5 +12,8 @@ How-to guides are recipes that take the user through steps in key subjects. They Contribute to Batavia's code Contribute to Batavia's documentation + Tour of the Code + Batavia's Test Suite Implementing Python Built-ins in JavaScript + Implementing Python Types in JavaScript Adding a module and testing it diff --git a/docs/how-to/tour.rst b/docs/how-to/tour.rst new file mode 100644 index 000000000..c7cceb39c --- /dev/null +++ b/docs/how-to/tour.rst @@ -0,0 +1,123 @@ +Tour of the Code +================ + +Before getting started, it's nice to know a bit about how the project is structured and where +to look for examples. This document aims to provide a brief tour of the +important features of the code. It's aimed at new contributors, but frequent flyers can +skip down to the bottom for a quick reference. + +General Structure +----------------- + +Core Code +********* + +Batavia implements the core language (types, builtins and the core interpreter) are all javascript, as well those +portions of the standard library that depend on system services. +This ensures quick execution and less compiling of the compiler. These are orgainzed into the +``builtins``, ``core``, ``modules``, ``stdlib``, and ``types`` files of the main ``/batavia`` directory, with +corresponding subdirectories. Alongside the virtual machine and test suite, this code makes +up the bulk of Batavia. New contributors should start with the ``types`` and ``builtins`` sections +for the best examples to review and copy from. These implementations are the foundation of Python as you know it and +should be immediately familiar to a Python developer. + +Support +******* +You'll also notice folders for tests, docs, and a few other sections, like ``testserver``, which is +a sample deployment with Django that allows you to test code in the browser. Contributions to the +tests and documentation are always welcome and are great ways to familiarize yourself with the +code and meet the other contributors. + +The Core Code +------------- + +A Word on Args & Kwargs +*********************** + +Batavia's implementations of various builtin functions +often **require** ``args`` and ``kwargs`` as input. Here's an example of calling +the repr of a ``my_thing`` object: ``builtins.repr(my_thing, [], {})`` + +The empty [] and {} arguments represent empty argument and keyword argument parameters. +This mimics how Python handles function arguments behind the scenes, and it's important! + +For instance, what happens when you pass a keyword argument into a list? You might say, +"list() doesn't take keyword arguments." In actuality, the list function does receive those +arguments, and the result is that it throws ``TypeError: '' is an invalid keyword +argument for this function`` + +Batavia needs those arguments explicitly specified in a standard format so that it can +check for that case and generate the correct error. The below code examples all use this calling +convention, and you'll be up to your knees in ``BataviaErrors`` if you're not aware of it. + +Building Blocks of Batavia +************************** + +This section is a quick reference for the most common code you'll see. + +builtins.js +^^^^^^^^^^^ + +.. code-block:: javascript + + var builtins = require('./builtins.js') + builtins.abs([-1], {}) // equivalent to abs(-1) + +This contains all of the native Python builtin functions, like ``str``, ``len``, and ``iter``. + +When dealing with Python types, many of the native JavaScript operations have been modified to +try to use builtins first. For instance, ``.toString()`` will often just call the object's ``__str__`` if +possible. Still, the best practice is to use the builtins and types wherever possible. + +types.js +^^^^^^^^ + +.. code-block:: javascript + + var types = require('./types.js) + var my_dict = new types.dict([], {}) + +This contains all of the native Python types that have been implemented in Batavia. It also has some helper functions: + +* ``types.js2py`` Converts a native JavaScript type to a corresponding Batavia type. +* ``types.isinstance`` checks to see if the object is a Python instance of the corresponding type. +* ``types.isbataviainstance`` checks to see if the object is an instance of **any** Batavia type. +* ``types.Type.type_name`` get the name of the type. + +This allows us to avoid ugly things like comparing ``Object.prototype.constructor``. Instead, +use ``types.isinstance`` or ``types.isbataviainstance``. Secondly, it's important that the inputs to Python +types are Pythonized themselves where needed. You should not be making a list() of JavaScript arrays, for +instance. That doesn't make sense! (It may even pass some tests, which is dangerous.) + +core/callables.js +^^^^^^^^^^^^^^^^^ + +These methods ensure that all Python code is executed using the proper ``__call__`` procedure, which could be +overriden or decorated by the programmer. + +* ``callables.call_function`` Invokes a function using its ``__call__`` method if possible. If not, just call it normally. +* ``callables.call_method`` Calls a class method using the call_function specification above. +* ``callables.iter_for_each`` Exhausts an iterable using the call_function specification above. + +As a general rule, use the builtin where possible. If no builtin is available, use the appropriate version +of ``call_function`` instead of calling Python functions and methods directly. An example: + +.. code-block:: javascript + + // Avoid this + my_thing.__len__() + + // Better + var callables = require('./core/callables.js') + callables.call_method(my_thing, '__len__', [], {}) + + // Best + var len = require('./builtins.js').len + len(my_thing, [], {}) + +Note the use of the Batavia calling convention in the two cases above! + +/core/version.js +^^^^^^^^^^^^^^^^ +Some helper functions for distinguishing the version of Python that's running. Outputs +vary from version to version, so it's nice to have this handy. \ No newline at end of file diff --git a/docs/how-to/types.rst b/docs/how-to/types.rst new file mode 100644 index 000000000..689b79578 --- /dev/null +++ b/docs/how-to/types.rst @@ -0,0 +1,152 @@ +Implementing Python Types +=========================================== + +Python's popularity is, in large part, due to the wonderful flexibility of its native types, like ``List`` and ``Dict``. In Batavia, Python native types are the building blocks for all of our other code. +This document will cover the structure of Batavia types and guide you on how to update the existing Batavia implementations. + +Process +------- + +The first thing to do when adding anything to Batavia is to play around a bit with it in the Python REPL. +Here's an example using ``int``:: + + >>> int + + >>> int(1) + 1 + >>> dir(int) + ['__abs__', '__add__', '__and__', '__bool__', '__ceil__', '__class__', '__delattr__', '__dir__', '__divmod__', '__doc__', '__eq__', '__float__', '__floor__', '__floordiv__', '__format__', '__ge__', '__getattribute__', '__getnewargs__', '__gt__', '__hash__', '__index__', '__init__', '__int__', '__invert__', '__le__', '__lshift__', '__lt__', '__mod__', '__mul__', '__ne__', '__neg__', '__new__', '__or__', '__pos__', '__pow__', '__radd__', '__rand__', '__rdivmod__', '__reduce__', '__reduce_ex__', '__repr__', '__rfloordiv__', '__rlshift__', '__rmod__', '__rmul__', '__ror__', '__round__', '__rpow__', '__rrshift__', '__rshift__', '__rsub__', '__rtruediv__', '__rxor__', '__setattr__', '__sizeof__', '__str__', '__sub__', '__subclasshook__', '__truediv__', '__trunc__', '__xor__', 'bit_length', 'conjugate', 'denominator', 'from_bytes', 'imag', 'numerator', 'real', 'to_bytes'] + +Your goal is to find out how the type responds to various inputs and outputs. You may also +want to consult the offical documentation. Once you're a little familiar, you can start to add your +implementation to Batavia. + +Anatomy of a Type +***************** + +Each Python type should be implemented as a JavaScript class. JavaScript handles classes similarly to Python, +but the syntax is very different. Each JavaScript class creates a constructor (similar to Python's ``__init__``), +which is any function that includes the ``this`` keyword, and a prototype, stored in ``.prototype``. +The prototype implements all of the methods for the class, including the constructor, and that's where we'll implement +the type's magic methods and other options. Let's take a look at ``List``. + +.. code-block:: javascript + + // Declare the list class. + function List() { + var builtins = require('../builtins') + + if (arguments.length === 0) { + this.push.apply(this) + } else if (arguments.length === 1) { + // Fast-path for native Array objects. + if (Array.isArray(arguments[0])) { + this.push.apply(this, arguments[0]) + } else { + var iterobj = builtins.iter([arguments[0]], null) + var self = this + callables.iter_for_each(iterobj, function(val) { + self.push(val) + }) + } + } else { + throw new exceptions.TypeError.$pyclass('list() takes at most 1 argument (' + arguments.length + ' given)') + } + } + + function Array_() {} + + Array_.prototype = [] + + List.prototype = Object.create(Array_.prototype) // Duplicates the prototype to avoid damaging the original + List.prototype.length = 0 + create_pyclass(List, 'list', true) // Register the class with Batavia + List.prototype.constructor = List + +This is the constructor, which is called by Batavia when someone invokes ``list()`` or ``[]``. We includes some code to inherit +JavaScript's native array prototype, which has much of the same functionality as List and has lots of quick functions. +You can use JavaScript natives in your implementation; this is a significant speed boost. + +Below that, you'll find all of the member methods added to the prototype. Note that each of these +should return a Python type from ``types.js``. + +.. code-block:: javascript + + List.prototype.__iter__ = function() { + return new ListIterator(this) + } + + List.prototype.__len__ = function() { + var types = require('../types') + return new types.Int(this.length) + } + +List also implements ``.toString()``, a JavaScript function that is sometimes called automatically when a string +conversion is needed. + +.. code-block:: javascript + + List.prototype.toString = function() { + return this.__str__() + } + +Note also the format for errors: ``throw new exceptions..$pyclass``. + +Making Changes +************** + +Make a Test +^^^^^^^^^^^ + +There is much work to be done in the types folder. When making changes, your goal is to match the output +of CPython and the output of the same call made in Batavia. Since we know the desired input and output, +we can use a test and then just fiddle. + +Head over to ``/tests`` and find the ``test_`` file. Many types have a robust test suite, but +do not assume that it is complete. +Follow the format there to add a test for your issue or modify an existing test. +Run it using the following command to check for errors. + +.. code-block:: bash + + $ python setup.py -s tests.path.to.your.test.TestClass.test_function + +Note: ``@expectedFailure`` indicates a test that's not passing yet. Your issue may be tested in one of those cases already. + +Pass the Test +^^^^^^^^^^^^^ + +If the test code runs and fails, you've identified the bug and should have some helpful output comparisons. Head over to +the type you want and start making edits, running your test until it passes. Occasionally, your bug will take you into +other Batavia types and builtins. That's fine too! There are a million places for small omissions all over the codebase. +Just keep in mind that the further you go down the rabbit hole, the more likely you are to have missed something simple. + +Once the test passes, run all tests for the class and see what else broke. (There's always something):: + + $ python setup.py -s tests.path.to.your.test + +After that, it's a good idea to pull the upstream master and check for merge conflicts.:: + + $ git add . + $ git commit -m "" + $ git fetch upstream + $ git rebase origin/master + +If you made major changes, then it may be a good idea to run the full test suite before submitting your pull request.:: + + $ python setup.py -s tests + +(Check out the sidebar for better/faster ways to run the full suite.) Finally, push your code to your fork and submit +your pull request on GitHub to run the CI. Fix any issues and push again until CI passes. The Batavia team will get back +to you with any additional notes and edits. + +Tips +^^^^ + +Your goal is to mimic the CPython implementation as much as possible. If you do so, you'll often fix multiple issues at once. Here's some tips: + +* The original implementation is documented in detail at https://docs.python.org/3/ -- reading up there will definitely improve your understanding. +* If you're inherting your class from JavaScript, which is very common, you get JavaScript's internal methods for free. Oftentimes, they can be left as is or lightly wrapped. +* Make sure your test properly covers the issue. For instance, if a member function accepts any iterable, make a generic iterable instead of using a list or tuple. +* Make sure method implementations accept args and kwargs, and throw appropriate errors if the input doesn't match. +* Keep your Python REPL open to the side and test your assumptions with lots of inputs and outputs.