Support pipes

[matthiasbeyer]

Would be nice if piping would be supported:

ls | grep foo

or provide some better interfaces for this:

ls.grep foo

[s-mage]

Is there a ruby way to do this?

 λ → home.ls # this would return String
 λ → home.ls.split("\n").select { |x| x =~ /Do/ }                                                                                                                                                                                  
["Documents", "Downloads"]   # this is how I would do it now.
λ → home.ls.e :grep, :foo                                                                                                                                                                                                         
sh: 2: bin: not found
sh: 3: chrome: not found
sh: 4: Desktop: not found
sh: 5: desktop.png: not found
sh: 6: Documents: not found
sh: 7: Downloads: not found
...
# This is what happened  when I exec grep on string :)

[Leo2807]

Maybe we could write a pipe function that takes a variable list of arguments.

[matthiasbeyer]

@Leo2807

A commandline call would then look like this:

pipe (pipe (pipe ls grep("foo")) grep("bar")) less

vs.

ls | grep foo | grep bar | less

What do you think now?

[s-mage]

I think it's more suitable to lisp :) Plus it's more ugly than in bash.
How pipes work in bash? There is stdout from one process that is adding to another as first or last argument?

[Leo2807]

Sorry if I was unclear, I meant a variable length argument list. It would look like this:

pipe :ls, :grep, :less

It would still need a way to add arguments though.

[matthiasbeyer]

@Leo2807

pipe ls: "-a" , grep: ["-E", /foo/], :less

Would be much better.

---

Okay, I'm stopping the trolling here now. This problem is rather hard to accomplish, yes. I would personally go like this:

1. Have a normal bash syntax for pipes (foo | bar)
2. Parse the commandline input into subcommands (AST parsing in its simplest form)
3. Call the commands and connect their stdin/stdout

So basically just what the bash does. We could prettify this with some neat keywords or something:

ls -> grep -E bar -> less
ls err> grep foo out> less

or whatever.

[s-mage]

http://stackoverflow.com/a/9834134/1685746 ah, it pushes that data to stdin. Oookay, I'll think how to implement that in ruby.

# for example
home.pipe :ls, %w(grep hello), :less
# this maybe work too
home.ls.pipe { |x| grep x "hello" }.pipe(&:less)

I'm not sure which one is better.

[s-mage]

Now I'm really thinking about how clojure deals with all that braces. They have threading macro that work
like this: https://clojuredocs.org/clojure.core/-%3E
What if we could do the same in ruby (but with some other name, since -> is lambda here.

home.ls.pipe { |x| grep x "hello" }.pipe(&:less)
# what if this would be equal
home.pipe_chain
  :ls,
  -> (x) { grep x "hello" },
  -> (x) { less '-R' x }

# no, it's ugly.

[RomanHargrave]

@s-mage I'm not sure what you are referring to by "braces". I assume you are referring to two different approaches to pipelining here:

1. Compositional form - chaining method calls to create a command object
2. Functional form - passing multiple command objects in the order they need to be piped

In my opinion, the first option would be superior, as a shell needs to allow piping more than just STDOUT, which the second notation could not allow for. This could easily be accounted for in the first by either modifying the pipe() method to support an optional 'fd' parameter for the file descriptor that is to be piped to the next process, or by adding another method 'pipefd(fd, command)' that supports specifying the file descriptor.

In terms of what fd should be, it would optimally be an integer for best compatibility with the underlying implementation.

[s-mage]

Hi, I'm back.

FYI, ruby supports pipelines via Open3 lib: http://ruby-doc.org/stdlib-2.0.0/libdoc/open3/rdoc/Open3.html

So one possible solution is just use Open3.pipeline, but it's not very convenient.

Another way is build my own pipeline based on popen3 (or maybe popen2). In the end it should look like this

home                 # box
  .ls('-lah')        # string
  .grep 'opensource' # string

[s-mage]

 λ → Open3.capture2("grep 'opensource'", stdin_data: home.ls('-lah'))                                                                                                                                                              
["drwxr-xr-x  44 s    s    4,0K окт.  16 18:20 opensource\n",
 #<Process::Status: pid 12124 exit 0>]

Example with Open3

[s-mage]

So I'd monkeypatch String class method_missing method, but it causes errors (I've just tried). Maybe I should create my own class and wrap every command output? Or is there more elegant solution?

[Leo2807]

I think that could work. Something like this:

class Pipe
    ...
    def method_missing(cmd, args*)
        ...
    end
end

pipe = Pipe.new someDir
print pipe.ls.grep('cow').hexdump

Or a more bash-like syntax:

class Pipe
    ...
    def |(pipe)
        ...
    end
end

ls = Pipe.new 'ls'
grep = Pipe.new 'grep'
hexdump = Pipe.new 'hexdump'

ls | grep('cow') | hexdump

[empjustine]

dir['*'].search(/foo/) seems to describe better what you want to do.

As a breaking change, home.ls should be an alias to home[*] IMHO.

The problem is that you're considering file names must be sane, and you can "simply newline split them", you're gonna have a bad time.

Using glob syntax and the Array it returns will (most of the time) free you from having to deal with those dangerous minutae.

If you don't, use at least ls -print0, and split using "\0", as good bash people should.

-print0
      True; print the full file name on the standard output, followed by a 
      null character (instead of the newline character that -print uses).  
      This allows file  names  that  contain  newlines or other types of white 
      space to be correctly interpreted by programs that process the find 
      output.  This option corresponds to the -0 option of xargs.

*not available on all flavours of ls such as busybox ls.