Permission to Point some people to my Repo - EXPERIMENTAL

[cryptonomicon314]

I added support for both Haskell and Clojure (Clojurescript support comes for free with Clojure, because they are almost identical), as well as support for automatically defined anchors in the source code.

I'd like to break backward compatibility in some places, and share my changes with actual Clojure programmers (I'm learning clojure in my free time) to see what they think. Would you mind if I pointed those people to my version while these changes are ready to merge into version 1.0?

The main features I added was support for automatic anchors to sections in which functions are defined. For example, sourcecode

(defn foo [bar]
  (...))

Adds the invisible anchor to the text part of the section

<a id="code.foo"></a>

So that you can refer to this function definition using the markdown extension [|foo @ filename-where-foo-is-defined|] or [|foo|] if foo is defined in the current file.

It is not very good to generate links to methods in a Python class, because it will generate repeated anchors if there are different classes with the same methods. I'll have to change it to go back to the parent section where the class name is defined.

The feature I would like to add is automatically hyperlinked source (with a decent amount of help by the user)

[ckald]

I was thinking about this feature. Personally, I'd prefer the combination of syntax of Markdown links we already use [[anchor|path/to/the/file#section]] and Sublime Text's goto-anything: path/to/the/file@function or path/to/the/file:linenumber.

So my preferred syntax will look like:

• [[path/to/the/file@function]]
• [[anchor|path/to/the/file@function]]

This does not require any new constructs, just another branch in replace_crossref function.

Because of the problem you mentioned, it is evident that we need a simplest algorithm for parsing a source file into a tree. That might be quite complicated to implement manually, but I'm pretty sure that we could reuse the modules used by the editors for autocompletion and go-to-definition features. I remember installing one massive library of this kind, but unfortunately I can't find it right away.

If we find that library, we can also use it's database to add lots of languages because it will definitely contain information about scope-defining statements.

Regarding the permission: of course you've got it, we are open source. And you can create a PR for your changes if you think they're ready

[ckald]

Might be useful (but not what I am looking for): https://pypi.python.org/pypi/CodeIntel

[ckald]

Here it is: http://ctags.sourceforge.net/ . It is tool that parses the source code an creates an index of objects. Does not support neither Haskell nor Clojure, but there's a workaround for the latter: http://clojurewise.blogspot.com/2010/03/using-ctags-with-clojure.html

Another interesting thingy: http://cedet.sourceforge.net/intellisense.shtml

[cryptonomicon314]

I've broken backward compatibility, but I guess it's worth it.
I've turned .pyccoon into .pyccoon.yaml, as we had discussed.

I've added support for hyperlinks in clojure files (not in my repo yet). It works using (dead-stupid) syntatic analysis only, but it works surprisingly well. The trick is to turn (perhaps qualified) names into urls that point either to local files or to external websites. Currently I support links to this website: https://crossclj.info/. This requires the user to define a mapping between module names and package names, but I include a big default map so that more common packages are supported out of the box.

The user can specify the mapping between namespaces and packages in the .pyccoon.yaml file, like this:

project:
    name: re-frame 0.4.1

files:
    # Note that we don't need to escape '\' in the strings:
    skip:
        - docs.*
        - \.git.*
        - .*/resources/public/js.*
    copy: []

documentation:
    css-path: null
    custom-html-template: null
    mathjax: true # should I make this the default? probably not...
    linebreaking-behavior: normal # this is now the default

language-specific:
    clojure:
        namespaces:
            # Local namespaces
            re-frame:
                # this method links to local files
                method: local
                # Relative to the directory on which
                # `pyccoon` is called.
                path: src/re_frame
                extension: cljs

            reagent.core:
                # this method links to the crosscljs.info website
                method: crossclj
                package: reagent
                is-cljs: true
                source: true

(actually, you don't need to mention reagent.core here, because it is included in the default mapping)

I had to turn .pyccoon into .pyccoon.yaml because otherwise the configuration would become unreadable. I'll document these options and try to implement equivalent functionality for python.

I'll upload my example project (a fork of the clojurescript re-frame library) so you can see what my version is capable of.

[cryptonomicon314]

I looked at the jedi autocomplete library for python, which looks promising, https://github.com/davidhalter/jedi but is probably overkill for our purposes.

[cryptonomicon314]

Just a catch: Do you know how to include pypandoc as an optional dependency? I really need it to parse haskell's documentation syntax (nothing else parses haddock comments), but this means the user must have the pandoc executable installed. Would it be possible to only install pypandoc if the user has installed pandoc?

[cryptonomicon314]

Example haskell project: http://cryptonomicon314.github.io/pyccoon-hs-xml/
This is the source of a standard haskell library.
I only made minor changes, such as making Headers compatible with markdown, because the library used headers like this:

-- Header here -----------------------
-- now a description
f :: a -> a -> b 

which I had to turn into headers like this:

-- ## Header here
-- now a description
f :: a -> a -> b 

Note the difference between how I handle haddock comments, which are meant to be API docs (here) and ordinary comments, for internal use (here)

[cryptonomicon314]

Example clojurescript project: http://cryptonomicon314.github.io/pyccoon-cljs-re-frame/
Again, I only changed the source slightly.

The source is hyperlinked both to external packages and internal modules. The links have no formatting, but the mouse cursor changes when on top of them. This is by design; I think hyperlinks should not interfere with normal syntax highlighting.

You can test the links in this file. In the namespace definition, the word reaction is an external hyperlink and the words warn and app-db are links to internal modules. This is all created automatically from the namespace definition and from the mapping between modules and packages specified in the .pyccoon.yaml file and the global namespaces database.

[ckald]

Haddock styles are a bit aggressive. Here is my 5 minutes take:

Hyperlinked source is done excellently — this is how it should be. But you need to be cautious, because implementing this kind of feature for any language can be completely outside the scope of this project. This is why I would like to use something like ctags that can already cover a lot of languages and is external to us.

I would like you to explain the namespaces in detail when you prepare a PR. And I like the yaml config file.

Regarding the custom headers styles: this is quite common, actually. We can add yet another setting that would allow use to insert his own regular expression to feed into a preprocessor that will turn all matches into corresponding Markdown-style headers. This will help to start with an existing project.

Regarding the catch: you can try this http://pythonhosted.org/setuptools/setuptools.html#declaring-extras-optional-features-with-their-own-dependencies

[cryptonomicon314]

Haddock styles are a bit aggressive.

I agree, I'm bad with CSS. I like your version better, but we need something that doesn't over the text. Maybe we can leave the "Haddock:" title out of this. Changing the background color can be enough (even though I like having a title that clearly explains that it is a Haddock comment).

We can add yet another setting that would allow use to insert his own regular expression to feed into a preprocessor that will turn all matches into corresponding Markdown-style headers

I disagree. We already have enough markdown extensions. I see no need to complicate them further for little benefit.

Implementing this kind of feature for any language can be completely outside the scope of this project

That's true, but there is nothing forbidding us to support some languages better than others.

[cryptonomicon314]

Actually, ctags is pretty cool and pygments supports it out of the box. The only problem is that I don't know if the data in ctags files will survive our funny business with "unique separators" and such. I'll try it when I have time to spare. The output off ctags for python files is disppointing, though, but that's a problem with python, which is unparseable with static tools (except for jedi, which does a good job), not with ctags itself. Actually, ctags is much smarter about python than my code is about clojure, but the way clojure code is structured makes it easy for my dead stupid algorithm to give good results.

I think that for most languages, support for hyperlinks will need a big rewrite... In the meantime, we have decent support for hyperlinked clojure files.

[ckald]

As I mentioned, you can try to use ctags with clojure using the Lisp mode. We can avoid the unique separator problem if we parse the whole file before processing.

But true, with the current approach it is hard to track line numbers of the source code. So many reasons to fix this and I have an idea how to do that. I think switching from linear parser into hierarchical (i.e., structuring sections of code and docs not in an list, but as a tree) will be productive

[cryptonomicon314]

I would like you to explain the namespaces in detail when you prepare a PR

There isn't much to explain - it's a clojure-specific thing... Namespaces are equivalent to python modules, and the namespaces part is just a map from namespace to package name with some extra options.

Actually, ctags is pretty cool and pygments supports it out of the box

I have implemented a über-cool version of my hyperlinking code, which is again unfortunately clojure specific. It handles both local and global vars, and the overwriting of a global var with a local var. Unlike ctags it handles external hyperlinks (and much better than my naïve code).

The best part is that it works by extracting the data into a sqlite database, which can be manipulated by python code common to all languages to create hyperlinked files. Having a real database of dependencies allows me to generate an index for identifiers used and defined by each module.

This architecture means that to add support for über-cool hyperlinks for a certain language, you only need to generate the relevant sqlite database, which works like a ctags file on steroids. Generating a complete and correct database is still very language specific, though.

The worst part is that this method of generating hyperlinks is much slower (1min30s for a project with 12 files). I'll have to profile the code to see what's exactly taking so long. Maybe it's inevitable given my architecture, in which case I should quit trying getting good hyperlinks into pyccoon.