feat(Directory): Add EnumLinksAsync method by hannahhoward · Pull Request #39 · ipfs/go-unixfs

hannahhoward · 2018-10-17T03:02:38Z

Goal

child of ipfs/kubo#5600
Implement an EnumLinksAsync method on Directory interface to support streaming version of ls command

Implementation

Add LinkResult type to unix-fs - is an ipld Link or an error
Add EnumLinksAsync method to Directory interface, returns channel of directory links or error
Add EnumLinksAsync method to Shard interface in HAMT, returns channel
of directory links or error
EnumLinks method in Shard interface in HAMT uses EnumLinksAsync now
modify makeAsyncTrieGetLinks to use channel

For Discussion

Arguably an async link enumeration is sufficiently generic it could live in go-ipld-format, and LinkResult is a type that is not super specific to go-unixfs
However I would prefer to hold on putting LinkResult in go-ipld-format to a later commit where we a generalized out-of-order parallel walker to IPLD format (essentially moving over enumerateChildrenAsync from go-merkledag)
Once this is merged, will move on to adding streaming ls to go-ipfs

schomatis

I'll take a closer look at the end of the week (left some comments for now) but this makes plenty of sense. 👍

hamt/hamt.go

unixfs.go

io/directory.go

hamt/hamt.go

magik6k · 2018-10-17T14:14:01Z

io/directory.go

+	go func() {
+		defer close(linkResults)
+		for _, l := range d.node.Links() {
+			linkResults <- format.LinkResult{


This should also select on ctx to avoid leaking goroutines (same in other places)

got it. will do.

magik6k

Few more nits, then LGTM

magik6k · 2018-10-19T13:00:00Z

hamt/hamt.go

-	format "github.com/ipfs/go-unixfs"
 	"github.com/spaolacci/murmur3"
+
+	format "github.com/ipfs/go-unixfs"


Should be above this group

unixfs.go

magik6k · 2018-10-19T13:19:15Z

io/directory.go


+// EnumLinksAsync returns a channel which will receive Links in the directory
+// as they are enumerated, where order is not gauranteed
+func (d *BasicDirectory) EnumLinksAsync(ctx context.Context) (<-chan format.LinkResult, error) {


I'd drop the error from the signature since it's not used here (or other impls of this function)

hamt/hamt.go

- Add LinkResult type to unix-fs - is an ipld Link or an error - Add EnumLinksAsync method to Directory interface, returns channel of directory links or error - Add EnumLinksAsync method to Shard interface in HAMT, returns channel of directory links or error - EnumLinks method in Shard interface in HAMT uses EnumLinksAsync now - modify makeAsyncTrieGetLinks to use channel

hannahhoward · 2018-10-30T00:39:08Z

@schomatis are you good with my responses? Can we go ahead and merge this PR?

schomatis

LGTM, I have one follow-up question but if this is blocking something else go ahead and merge.

magik6k

LGTM module the @schomatis comment (not blocking for me)

hannahhoward · 2018-10-30T19:39:00Z

I think I'm going to go ahead and merge but feel free to continue this discussion.

…5600 feat(Directory): Add EnumLinksAsync method This commit was moved from ipfs/go-unixfs@a3eae7f

hannahhoward requested a review from schomatis as a code owner October 17, 2018 03:02

ghost assigned hannahhoward Oct 17, 2018

ghost added the status/in-progress In progress label Oct 17, 2018

hannahhoward requested review from eingenito and michaelavila October 17, 2018 03:04

schomatis reviewed Oct 17, 2018

View reviewed changes

hamt/hamt.go Show resolved Hide resolved

unixfs.go Show resolved Hide resolved

io/directory.go Outdated Show resolved Hide resolved

hamt/hamt.go Outdated Show resolved Hide resolved

hannahhoward requested a review from magik6k October 17, 2018 13:55

magik6k reviewed Oct 17, 2018

View reviewed changes

hannahhoward force-pushed the features/streaming-ls-5600 branch 2 times, most recently from cfc13dc to 6ec34f8 Compare October 18, 2018 14:47

hannahhoward mentioned this pull request Oct 18, 2018

Add --stream option to ls command ipfs/kubo#5611

Merged

Stebalien requested a review from magik6k October 18, 2018 15:53

magik6k reviewed Oct 19, 2018

View reviewed changes

schomatis reviewed Oct 29, 2018

View reviewed changes

hamt/hamt.go Show resolved Hide resolved

schomatis reviewed Oct 29, 2018

View reviewed changes

hamt/hamt.go Show resolved Hide resolved

hannahhoward added 6 commits October 29, 2018 15:06

Add context cancelling logic

8701219

Convert formatting in hamt links to remove bit prefix

fa995d3

Force processing of context cancellation first

a5528c4

Emit errors once at end

269f6d2

feat(EnumLinksAsync): Remove error since it's unused

c54d0e4

hannahhoward force-pushed the features/streaming-ls-5600 branch from eee76dd to c54d0e4 Compare October 29, 2018 22:06

schomatis approved these changes Oct 30, 2018

View reviewed changes

magik6k approved these changes Oct 30, 2018

View reviewed changes

hannahhoward mentioned this pull request Oct 30, 2018

commands/ls: use new cmds lib ipfs/kubo#5663

Closed

hannahhoward merged commit a3eae7f into master Oct 30, 2018

ghost removed the status/in-progress In progress label Oct 30, 2018

hannahhoward mentioned this pull request Nov 8, 2018

Wait for all go routines to finish before function returns ipfs/go-merkledag#19

Merged

Stebalien deleted the features/streaming-ls-5600 branch November 8, 2018 22:41

Jorropo pushed a commit to Jorropo/go-libipfs that referenced this pull request Jan 25, 2023

Merge pull request ipfs/go-unixfs#39 from ipfs/features/streaming-ls-…

05c5c66

…5600 feat(Directory): Add EnumLinksAsync method This commit was moved from ipfs/go-unixfs@a3eae7f

Conversation

hannahhoward commented Oct 17, 2018

Goal

Implementation

For Discussion

Uh oh!

schomatis left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

magik6k Oct 17, 2018

Choose a reason for hiding this comment

Uh oh!

hannahhoward Oct 17, 2018

Choose a reason for hiding this comment

Uh oh!

magik6k left a comment

Choose a reason for hiding this comment

Uh oh!

magik6k Oct 19, 2018

Choose a reason for hiding this comment

Uh oh!

Uh oh!

magik6k Oct 19, 2018

Choose a reason for hiding this comment

Uh oh!

hannahhoward Oct 29, 2018

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

hannahhoward commented Oct 30, 2018

Uh oh!

schomatis left a comment

Choose a reason for hiding this comment

Uh oh!

magik6k left a comment

Choose a reason for hiding this comment

Uh oh!

hannahhoward commented Oct 30, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants