Why ask for array support without specifying primitives?

[spion]

I had some questions that I asked Raynos today about yielding arrays - I didn't understand why the spec has a section about that.

The answer was that:

Yielding arrays is done to enable a parallel primitive without dependending on a third party library - this would allow any arbitrary generator on npm to use parallel execution without having to ask himself "is the AGen compliant runner going to support my parallel thing" ?

Thats a great goal, but if the spec doesn't specify the minimum supported types of asynchrony. This means that the answer to the question

"is the AGen compliant runner going to support my parallel thing" ?

is always: I have no idea - what kind of asynchrony does it use?

For all we know, the AGen compliant runner only works with Reactive Extensions primitives, and your published generator relies on yielding promises. And thats the reason why I'm not getting the benefit of specifying behavior when yielding arrays... The primitives may already be incompatible, so why specify the behavior of arrays?

Another point is that once you change the type of asynchrony, the method to achieve parallelism may also change. As an example, genny doesn't even have an asynchrony type - it yields emptiness most of the time. And it achieves parallelism by running operations without yielding, then yielding the resumer function the same number of times. Same thing applies to promises too - arrays are completely unnecessary to achieve parallelism there. You can just start the operations in parallel and get promises, then loop through all the promises to yield them.

The only remaining situation where yielding arrays is really beneficial are thunks...

So my question kind of is, what goal does this spec aim to achieve? Am I missing something obvious here?

[CrabDude]

The goal of the spec is to specify the minimum necessary interface to facilitate interoperability. If you decided to use a library that used Reactive Extensions primitives, you could optionally adopt an implementation that supported your asynchrony type without changing any other code. That last phrase is the key understanding for why we built this spec.

However, you raise an interesting point. The spec should definitely include erring on unsupported asynchrony to avoid silent failures like you suggest.

[spion]

That was not my original question / point, but thats okay - I clarified the meaning of arrays. Although another point that I raised, that the means to achieve parallelism may change depending on the asynchrony type - still remains.

Which brings me to the next question, namely:

yield [(function*(){})(), asynchrony]

if I interpret correctly, at the same time wait for yield* generatorObject and yield asynchrony

Can you point me to a library that implements this for both thunks and promises? For example, at first glance except for the sugar and agen compatibility, it looks a bit unnecessary for promises which more often than not have their own composition functions already - so promise implementers may complain a bit.

[CrabDude]

A primary motivation for this spec is to stop having to use promises or thunks for anything except core IO, and even then for it to be extremely similar, e.g.,

yield fs.readFile.bind(fs, 'data.txt')
// vs
yield q.nfcall(fs.readFile, 'data.txt')

@Raynos' gens and @visionmedia's co both support yielding on arrays of thunks or promises in parallel and @creationix's gen-run supports arrays of thunks. In addition, your own genny supports yielding arrays of thunks and promises, but via an accumulator style.

So if I understand correctly, your point is, "Why is support for yielding on an array necessary if we can support equivalent functionality via an accumulator similar to genny's?" and the response is, "compatibillity." This is precisely the goal of AGen, to specify an interoperable pattern without being opinionated on Asynchrony types.

[Raynos]

"is the AGen compliant runner going to support my parallel thing" ?

It is. Because parallel is yield (Array<Asynchrony> | Array<GeneratorObject>). That is how you execute things in parallel.

For all we know, the AGen compliant runner only works with Reactive Extensions primitives, and your published generator relies on yielding promises.

I think we should specify a minimum set of Asynchrony that is { thunks, promises } to avoid this problem.

However note that you can have a generator function that works with every asynchrony

var path = require("path")

function* mkdirp(mkdir, path) {
  var tuple = yield { both: mkdir(path) }
  if (!tuple[0]) {
    return true
  }

  yield* mkdirp(mkdir, path.dirname(path))
  yield* mkdirp(mkdir, path)
}

Here we pass in an implementation of mkdir that returns some arbitary asynchrony. This entire generator function is asynchrony agnostic.

The primitives may already be incompatible, so why specify the behavior of arrays?

Because if your generator is primitive agnostic it doesn't know whether to use require("continuable-para") or require("bluebird").All for parallel execution. However it knows that if it yields an array then the runner will use the correct primitive.

But more importantly, let's say we stick to generator functions of continuables & promises for now. If the spec does not allow parallel yielding with [ ... ] then what happens is that every generator function written by an author will use that authors favorite parallel execution flow control library, of which there are thousands.

Having a standardized way of doing very common async operations means that you can jump into other peoples code without learning how that flow control library they wrote works.

And it achieves parallelism by running operations without yielding

That whole idea is great, I havn't thought about it. However that's not how AGens works :(. We can't support that. But genny can, however modules written in that style will break in co & gens. So that style is great for app development but not for npm module development.

[Raynos]

So my question kind of is, what goal does this spec aim to achieve? Am I missing something obvious here?

This should be in the readme.

Motivation.

Currently if you want to author a module that exports a generator function you have two option. You either claim it works with an existing generator runner like co or you wrap it in the generator runner before you export it (thus turning it into a normal callback / promise / whatever function).

Ideally what you want to do is just export a generator function and have other authors yield* your function. For this to work you need to make your generator function comply with some minimal set of functionality that all runners should adhere to, otherwise your generator functions is tightly coupled to co or gens or genny.

So this spec defines some set of functionality including parallel & array execution. This is because you may want to take arbitrary asynchronies as arguments and delegate to the runners parallel execution instead of take promises / callbacks as arguments etc.

It also means that you can define generator functions that dont have a dependency on a generator runner or custom flow control library for 95% of the use cases.

[Raynos]

@spion

yield [(function*(){})(), asynchrony]

I have not implemented that. The spec needs to be more clear.

As far as I understand currently this is valid:

type Asynchrony := Continuable<T> | Promise<T>

type YieldExpression<T> := 
  Asynchrony<T>

type YieldExpression<Array<T>> :=   
  Array<Asynchrony<T>> |
  Array<GeneratorObject<T>>

type YieldExpression<[Error, T]> :=
  { both: Asynchrony<T> | GeneratorObject<T> }

var T = yield YieldExpression<T>

i.e. those are the types you can yield and the types they return.

Which means mixing & matching generator objects & asychronies is not valid.

Further more the spec also says you should be able to return Error objects which should do the correct thing like be interceptable by { both }

[creationix]

Ok, so I think I'm starting to understand the goal here. You don't care if the actual yielded value is a continuable, promise, or something else because you also don't care if the author chooses to use gen-run, co, or something else.

This spec is for the library in the middle that doesn't create it's own continuables/promises and doesn't depend on an async runner. It just uses yield, yield* and [] as if they were language natives. Is this right?

This means you still can't mix and match libraries if they produce different continuable/promise values themselves. Only if they yield* only whatever they are given by the underlying library.

There is value in such an interop spec, but not near as much value as if everyone also agreed on the actual value to yield.

[Raynos]

@creationix there is indeed value in such an interop spec.

However there is ALSO value in an interop spec that says all these things and that all asynchronities must be continuables or promises.

This means that you can publish functions to npm

• that yield on arbitrary asynchronies as arguments
• that yield on promises
• that yield on thunks

It also means that you can choose to use Reactive Extensions in private app code and have a generator runner that does thunk/continuable + promise + reactive extensions in your own app BUT still use functions from npm that just work with thunks / promises.

I wanted to avoid telling people to only use thunks / promises because that leads to land of opinion & bike shed.

TL:DR; We should add minimum support for promises & thunks to all generator runners so that as long as we stick with those two for public npm modules no problems occur.

[creationix]

Right, if you add to the spec that async values must be either promises or continuables and all async runners support both (it's not hard to support both), then it will have a lot of interop. I could get on board with that and add promise support to gen-run.

[CrabDude]

See #6 for a proposal to include promise and continuable support in the spec.