Do we need more functional elements?

[timyates]

So currently you can do:

def s = Stream.from x:1..3, y:1..3 map { x + y } filter { x != y }
assert s.take( 5 ).collect() == [ 3, 4, 3, 5, 4 ]

And to stop a stream you can do:

def s = Stream.from { x + y++ } filter { it < 5 ?: STOP } using x:1, y:1
assert s.take( 5 ).collect() == [ 2, 3, 4 ]

But I need to think whether it's worth adding some different constructs instead, something like:

def s = Stream.using x:1, y:1 map { x + y } update { x += 1 ; y += 3 } until { x >= 3 }
assert s.take( 5 ).collect() == [ 2, 6 ]

Anyone got any opinions before I arbitrarily make a decision?

Knowing me, you've got a good few weeks ;-)

[timyates]

Added an until block as of commit d715f7c

[tbarker9]

What about validate{}? It would work like filter, but also filter out anything causes an AssertionError.

[timyates]

So you'd do:

Stream.from list validate { assert it % 2 == 0 }

Rather than

Stream.from list filter { it % 2 == 0 }

?

What does it gain you?

[tbarker9]

Might be nice for Stream to optionally store some stats. Four counts might be enough:

processed - count of all items that went through all steps without be filtered out
ignored - count of all items that were filtered out (not including invalid)
invalid - count of all items that were invalid
total - processed + ignored + invalid

[tbarker9]

More descriptive I suppose. Say I am processing a list of people with some dirty data. I only want males (filter), but say one of the people has a number for a last name, I would put that check in validate.

[timyates]

Yeah, that would be nice... Need to think where this data can be stored...

I was thinking the other week, if we get rid of the using block, then the whole stream can be implemented as a chain of Iterators. However this does mean that each iterator step will be ignorant of the other steps in the chain, so metrics like this would be tricky (unless the stream keeps it's parent steps, and allows the user to interrogate each step)

Still not decided in my head whether it's worth abandoning using though... It's quite nice to have variables scoped to the stream, but it abandons all sense of functional purity

[tbarker9]

Although with that example I could still use just conditionals and have validate match filter functionality. Given that a lot of people might use this library for data processing (me 😄 ), having validate regardless of whether or not it supports AssertErrors might be handy. Think of validate as given vs setup in spock. Its existence might be for readability only.

[tbarker9]

To get the stats to work, you could implement each or something like each and return the stats. Maybe be explicit about it with eachWithStats.

[tbarker9]

Actually, after looking at Stream, why can't you put the stats in there? You already iterate over the steps in loadNext. Probably a good idea to put the stats in a class of its own.

[timyates]

Added the stats as issue #14 so I don't forget or lose this 😉

[tbarker9]

This is probably totally out of scope, but might as well shoot for the stars! What about chunking? Say you have

Stream.from 1..8 chunk(5)

This will create a list with two element, each containing a list of their own. The first element contains 1 to 5, the second 6 to 8. Each chunk has no more than 5 elements. Pretty handy if you want to process data to a database in transactional chunks. aggregate might be a reasonable name as well. To make it easier to implement, you could make this a terminating step where a plain old iterator is returned. Otherwise dealing with stats gets really dicy.....

[timyates]

I like it, kinda the opposite of flatMap?

I think chunk is the name used in Scala as well (and it's always good to
stick with used names)

Or was that chunkBy? (that's probably a different method though)

I'll do some searching... Can you add this as a new issue?

[tbarker9]

I have a batch iterator lying around somewhere that can probably do this. Basically wraps another iterator and returns 'chunks'. Let me see if I can use it here.

[tbarker9]

I like the idea of making the stream a chain of iterators. If someone wanted to make a stream 'plugin', you could do it easily using a groovy extension. You could also have a method addStep to add a custom step.