Infinite series expansion

[cyclotron3k]

parallel/lib/parallel.rb

Line 92 in 4fe8381

@source = source.to_a unless @lambda # turn Range and other Enumerable-s into an Array

Parallel provides each and map methods, implying it works well with Enumerables, but it fails with the following [contrived] use case:

require 'parallel'
require 'prime'

primes = Prime.to_enum
private_key = 12344567899

Parallel.each(primes, in_threads: 16) do |prime|
  if private_key % prime == 0
    puts "Found factor: #{prime}"
    raise Parallel::Break
  end
end

...because if the first argument to Parallel.each is not a lambda, .to_a is called and it tries to expand the infinite series.

[grosser]

so basically you are saying it should use only the enumerable interface ?

The array interface has the advantage of not needing to send all elements across the inter-process-pipe which means they don't get serialized (overhead / possibly unserializable), so I'd like to avoid that when possible.

For this example a possible solution would be to create a lambda that yields primes.next.

A Range would be in the same territory and is initially what prompted the .to_a addition, so idk a failsafe way to automatically decide if to_a should be used or not.

A solution could be to say "if it is not an Array but an Enumerable use the Enumerable interface". That would also solve infinite ranges like 1..Float::INFINITY and would still be cheap enough to send over the inter-process-pipe.

I kinda doubt it's a big issue since it never came up before and there are easy workarounds, so I'd prefer to not change the internal behavior for no good reason. If you want to experiment with it, make a PR and see how messy it gets and maybe ping a few prior contributors on their thoughts.

[cyclotron3k]

I've just been using Enumerator a bit recently, and it would be useful (to me) if it could be lazily evaluated in the same way that Parallel handles lambdas. It would prevent the use of progress, though, which would be an issue.

I'll have a play and see if I can come up with anything useful/clean.

[grosser]

ah true, progress would break too :( might just make it easier by adding a `enum: true` option that does the enum->lambda conversion ... but idk if that's a lot of help

…

On Mon, Oct 23, 2017 at 12:23 AM, cyclotron3k ***@***.***> wrote: I've just been using Enumerator a bit recently, and it would be useful (to me) if it could be lazily evaluated in the same way that Parallel handles lambdas. It would prevent the use of progress, though, which would be an issue. I'll have a play and see if I can come up with anything useful/clean. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#211 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAsZ58Hx75jFlL8DjMpCsFp0lt3XGoQks5su8BrgaJpZM4P9DR7> .