User Tools

Site Tools


extpgms

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

extpgms [2018/12/27 13:27] (current)
Line 1: Line 1:
 +
 +~~CLOSETOC~~
 +
 +~~TOC 1-3 wide~~
 +
 +```juliarepl
 +julia> pkgchk.( [ "​julia"​ => v"​1.0.3"​ ] );
 +```
 +
 +
 +# Executing External Programs (and Capturing Output)
 +
 +## Running and Capturing Single OS Programs
 +
 +Julia can run arbitrary external programs, such as shell scripts, from within the [[installation|REPL]]. The backtick notation is used to represent commands, although it is not used to run the command (unlike in many other scripting languages).
 +
 +```juliarepl
 +julia> mycommand= `echo "​unix"​` ​   ## do not run but define. ​ PS: backquoted allow interpolation
 +`echo unix`
 +
 +julia> typeof( mycommand )        ## mycommand is *not* a string
 +Cmd
 +
 +julia> run( mycommand )           ## now run, but do not capture
 +unix
 +Process(`echo unix`, ProcessExited(0))
 +
 +julia> a= read( mycommand, String ) ## run and capture
 +"​unix\n"​
 +
 +julia> a                          ## captured
 +"​unix\n"​
 +
 +```
 +
 +To write to unix commands:
 +
 +```juliarepl
 +julia> open(`grep -n ''​`,​ "​w",​ stdout ) do ofile; ​ for i=0:2; println(ofile,​ '​A'​+i);​ end;   ​end#​do
 +1:A
 +2:B
 +3:C
 +```
 +
 +
 +## Pipelines
 +
 +Julia does not pass the command string to the shell for interpretation. ​ Instead, Julia rewrote shell functionality.
 +
 +By intention, Julia does not allow passing entire shell sequences. ​ For example, you cannot "run( ps | cut -c26-100 )". (If you absolutely need the shell escape, you can always write the shell command to a shell script and then run the shell script.) ​  ​Instead,​ Julia offers something better, which
 +
 +* reduces inadvertent shell-escape security holes, and
 +
 +* provides better error messages about which part of a pipe has failed in a pipeline (when it does).
 +
 +
 +## Opening, Reading/​Writing,​ and Closing Pipelines
 +
 +### Writing To a Pipe
 +
 +```juliarepl
 +julia> ​ open( pipeline(`gzip -c`, stdout="/​tmp/​pipelined.txt.gz"​),​ "​w"​ ) do opipe;
 +     println(opipe,​ "this is gzip encoded,​\nand here is our next line.\n"​)
 + end#do##
 +```
 +
 +FIXME Because this pipeline is really only one stage, this can also be written as `open( `gzip -c`, "/​tmp/​pipelined.txt.gz",​ "​w"​ )` ?!
 +
 +### Reading From a Pipe
 +
 +```juliarepl
 +julia> ​ open( pipeline(`gzcat /​tmp/​pipelined.txt.gz`),​ "​r"​ ) do ipipe;
 +             ​readlines(ipipe)
 + end#do##
 +3-element Array{String,​1}:​
 +"this is gzip encoded,"​
 +"and here is our next line."
 + ""​
 +
 +```
 +
 +### Reading from a Multi-State Pipe
 +
 +```juliarepl
 +julia> p= pipeline(`ps -ax` , `grep /​sbin/​launchd` , `grep ' 1 '` , `cut -c1-10,​26-100`)
 +pipeline(pipeline(pipeline(`ps -ax`, stdout=`grep /​sbin/​launchd`),​ stdout=`grep ' 1 '`), stdout=`cut -c1-10,​26-100`)
 +
 +julia> run(p);
 +    1 ??  /​sbin/​launchd
 +```
 +
 +PS: Note how the `stdout=` was automatically added to `p`.  You could also provide destinations explicity, or even redirect stdin, stdout, and/or stderr.
 +
 +See also [Running External Commands](https://​docs.julialang.org/​en/​stable/​manual/​running-external-programs/​).
 +
 +FIXME mention that '&'​ can be used to simultaneously run multiple shell commands, too!
 +
 +
 +### Accessing File and Process Components of The Pipeline
 +
 +```juliarepl
 +julia> pin= open( pipeline(`gzcat /​tmp/​pipelined.txt.gz`),​ "​r"​ )    ## alternative reader with more detail info
 +Process(`gzcat /​tmp/​pipelined.txt.gz`,​ ProcessExited(0))
 +
 +julia> readline( pin )       ## note: pin is Process <: IO object, so it can use IO functionality
 +"this is gzip encoded,"​
 +
 +julia> readline( pin )
 +"and here is our next line."
 +
 +julia> readline( pin )
 +""​
 +
 +julia> readline( pin )       ## reading beyond the file end returns empty, and not an exception
 +""​
 +
 +julia> close( pin )
 +
 +```
 +
 +* as always, `readchomp()` eliminates the final \n (but not intermediate ones).
 +
 +
 +
 +
 +
 +
 +## Controlling Both Input and Output of External Programs or Sockets
 +
 +Be careful...be very very careful. ​ You may be waiting for program output to continue---but your program may be waiting for input from you.
 +
 +
 +### Other Unix Programs
 +
 +FIXME create an example that 'feeds to unix dc' and picks off the output.
 +
 +feed into dc: 100 200 + f\n150 - f\n
 +obtain:
 +
 +```juliafix
 +writer = @async writeall(`cal`,​ "12 2000") ;;
 +reader = @async do_compute(reads(`sort,​ String`))
 +wait(process)
 +fetch(reader)
 +```
 +
 +#### THIS NO LONGER WORKS
 +
 +QUESTION FIXME Tell me how to do this.
 +
 +```julianoeval
 +julia> (so,si,pr)= readandwrite(`cat`)
 +(Pipe(active,​ 0 bytes waiting),​Pipe(open,​ 0 bytes waiting),​Process(`cat`,​ ProcessRunning))
 +
 +julia> write(si,"​hello\ngood\nbye\n"​)
 +15
 +
 +julia> so
 +Pipe(active,​ 15 bytes waiting)
 +
 +julia> si
 +Pipe(open, 0 bytes waiting)
 +
 +julia> close(si)
 +
 +julia> so
 +Pipe(closed,​ 15 bytes waiting)
 +
 +julia> pr
 +Process(`cat`,​ ProcessExited(0))
 +
 +julia> readall(so)
 +"​hello\ngood\nbye\n"​
 +
 +julia> so
 +Pipe(closed,​ 0 bytes waiting)
 +```
 +
 +
 +
 +
 +
 +
 +### Network Socket
 +
 +Adopted from [Networking](https://​docs.julialang.org/​en/​stable/​manual/​networking-and-streams/#​A-simple-TCP-example-1)
 +
 +FIXME the network socket docs were wrong in 0.6
 +
 +listen, connect, getaddrinfo,​ etc.
 +
 +listen(2001) ​ == localhost:​2000; ​ listen(IPv4(0),​ 2001) is external; ​ listen("​mysocket"​) is a PipeServer.
 +
 +```juliafix
 +julia> using Sockets
 +
 +julia> @async begin
 +            server = listen(2000)
 +            while true
 +            while true
 +                sock = accept(server)
 +                @async while isopen(sock)
 +                   ​write(sock,​ readline(sock,​ keep=true))
 +                end
 +            end
 +        end##​begin##​
 + Task (runnable) @0x00007fd31dc12e60
 +
 +julia> clientside = connect(2001)
 + ​TCPSocket(RawFD(28) open, 0 bytes waiting)
 + 
 +julia> @async while isopen(clientside)
 +           ​write(stdout,​ readline(clientside,​ keep=true))
 +        end;##​while##​
 +Task (runnable) @0x00007fd31dc11870
 +
 +```
 +
 +#### VERY OLD??
 +
 +```juliafix
 +julia> mysocket= sock ;
 +       ​myserver=
 +           ​@async begin
 +           ​server = listen(2001) ​  ## == localhost:​2000;​
 +           while true
 +               ​global sock = accept(server)
 +               ​@async while isopen(sock)
 +                   ​rcvmsg= readline(sock)
 +                   ​(rcvmsg == "​end\n"​) && break
 +                   ​write(sock,"​received '​$rcvmsg'​\n"​)
 +               end
 +           end
 +       ​end##​end##​
 +Task (runnable) @0x00007fd31dc12e60
 +
 +julia> clientside = connect(2001)
 +TCPSocket(RawFD(28) open, 0 bytes waiting)
 +
 +julia> @async while true
 +           ​write(stdout,​readline(clientside))
 +       end
 +Task (runnable) @0x00007fd31dc11870
 +
 +julia> println(clientside,"​Hello World from the Echo Server"​)
 +Hello World from the Echo Server
 +
 +julia> close(clientside)
 +
 +```
 +
 +FIXME how to close the network socket server?
 +
 +FIXME what does `readavailable()` do?
 +
 +
 +
 +
 +# Backmatter
 +
 +## Useful Packages on Julia Repository
 +
 +## Notes
 +
 +* There is also `mmap()` that makes it possible to designate memory as an IOStream.
 +
 +* A wrapper that makes command calling simpler (but worse) is https://​github.com/​innerlee/​Shell.jl .
 +
 +## References
 +
 +* [Shelling Out Sucks](https://​julialang.org/​blog/​2012/​03/​shelling-out-sucks) and [Put this in Your Pipe](https://​julialang.org/​blog/​2013/​04/​put-this-in-your-pipe).
 +
 +* http://​blog.leahhanson.us/​post/​julia/​julia-commands.html
  
extpgms.txt ยท Last modified: 2018/12/27 13:27 (external edit)