User Tools

Site Tools


programming

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

programming [2019/01/02 11:39] (current)
Line 1: Line 1:
 +
 +~~CLOSETOC~~
 +
 +~~TOC 1-3 wide~~
 +
 +
 +```juliarepl
 +julia> pkgchk.( [ "​julia"​ => v"​1.0.3",​ "​TimerOutputs"​ => v"​0.4.0"​ ] );
 +
 +```
 +
 +
 +# Advanced Programming
 +
 +See also the chapters on [[fundispatch]] and [[funother]].
 +
 +* Benchmarking and Unit-Testing work reasonably well.
 +
 +* Debugging and profiling are rudimentary in 0.6.2.  ​
 +
 +
 +
 +## Coding Style
 +
 +### Julia Official
 +
 +The Julia developers request the code style to be
 +
 +* lowercase variable, function, and macro names,
 +* capitalize Types and Modules,
 +* avoid underscores (CamelCase preferred).
 +
 +The full style guide is [here](https://​docs.julialang.org/​en/​v1/​manual/​style-guide/​).
 +
 +### Cookbook Inofficial
 +
 +I request that code be written to do the following
 +
 +* make all variables `const` as soon as possible
 +
 +* explicitly (strongly) type all function inputs and outputs
 +
 +* use the @boundscheck macro liberally inside functions with in-code tests.
 +
 +
 +
 +
 +## Type Mutability
 +
 +It is generally better to avoid type mutability (i.e., the fact that the same variable may change type). ​ For example,
 +
 +```julianoeval
 +julia> function summer(A)
 +                  s = 0
 +                  for a in A
 +                      s += a
 +                  end#for#
 +                  return s
 +              end;##​function##​
 +
 +```
 +
 +could have `s` change type (from Int to Float) if called with `summer([1.0,​2.0,​3.0])`. ​ You can avoid this type change with `s=zero(eltype(A))`. ​ To diagnose type mutability, invoke from the REPL
 +
 +```julianoeval
 +julia> @code_warntype summer([1.0,​2.0,​3.0])
 +
 +```
 +
 +and look for "​red"​ output.
 +
 +
 +
 +
 +## Programming Projects With Scaffolding
 +
 +The [manual](https://​docs.julialang.org/​en/​v1/​stdlib/​Pkg/#​Creating-your-own-packages-1) describes the process in more detail.
 +
 +### Generation
 +
 +```julianoeval
 +julia> using Pkg
 +
 +julia> Pkg.generate("​MyFirstPackage"​)
 +Generating project MyFirstProject:​
 +    MyFirstProject/​Project.toml
 +    MyFirstProject/​src/​MyFirstProject.jl
 +```
 +
 +The `MyFirstPackage.toml` file is
 +
 +```julia
 +authors = ["ivo welch <​ivo.welch@gmail.com>"​]
 +name = "​MyFirstProject"​
 +uuid = "​73461ff8-ab02-11e8-1d84-2180feccd51a"​
 +version = "​0.1.0"​
 +
 +[deps]
 +```
 +
 +There are elaborate ways to declare dependencies to other packages, all explained in the manual.
 +
 +
 +### Modules
 +
 +The `src/​MyFirstProject.jl` file is a module in a jl file.  The default is
 +
 +```julia
 +module MyFirstProject
 +
 +greet() = print("​Hello World!"​)
 +
 +end # module
 +```
 +
 +Please see the extended description below.
 +
 +
 +### Running
 +
 +```julianoeval
 +julia> Pkg.activate("​./​MyFirstProject"​) ##​ where to find the project.
 +"/​Users/​ivo/​MyFirstProject/​Project.toml"​
 +
 +julia> import MyFirstProject ##​ load the project. ​ using = import
 +[ Info: Precompiling MyFirstProject [73461ff8-ab02-11e8-1d84-2180feccd51a]
 +
 +julia> MyFirstProject.greet() ##​ run the project
 +Hello World!
 +
 +```
 +
 +### Add tests to the project
 +
 +```julianoeval
 +julia> print( open("​MyFirstProject/​test/​runtests.jl",​ "​w"​),​ "​println(\"​Testing...\"​)"​ ); gc();
 +
 +julia> Pkg.test("​MyFirstProject"​)
 +Testing...
 +   ​Testing MyFirstProject tests passed
 +
 +```
 +
 +
 +
 +### Changing the LOAD_PATH
 +
 +When a package is included, julia automatically knows that other included .jl files can also be in the package src directory. ​ this makes including them seamless.
 +
 +Nevertheless,​ it is sometimes useful to add to the loadpath, so that related packages and files can be found. ​ This can be done with `push!(LOAD_PATH,​ "​./"​)`.
 +
 +
 +### Install Your Package For Your Own Use
 +
 +* Packages for one project can be kept in the local directory, which is activated with `Pkg.activate("​."​)`.
 +
 +* The default location for user packages that should be available systemwide (i.e. from everywhere and without a mandatory `Pkg.activate`) is FIXME.
 +
 +
 +### Github Storage
 +
 +Docs, Chapter 35.
 +
 +```sh
 +$ git config --global user.name "Me Is"
 +$ git config --global user.email "​email@gmail.com"​
 +
 +```
 +
 +
 +QUESTION FIXME Any Special Support for Git?  Or do we just use the standard CLI from the Unix shell?
 +
 +
 +
 +### Contribute Your Package to the Repository
 +
 +FIXME `PkgDev` is broken in Julia 1.0 as of Sep 2018
 +
 +Also, `PkgDev.register("​FooBar"​)` and  `PkgDev.publish()`.
 +
 +
 +
 +### Extended Discussion of Modules For Packages
 +
 +Modules (namespaces) are really designed for use in packages. ​ They have import/​export and variable protection facilities, but when used as part of an ordinary program (i.e., just included with *.jl), they loose all these features. ​ Moreover, in this case, you can only access their variables in fully qualified form (i.e., modulename.variablename).
 +
 +If you `Pkg.generate('​aha'​)` and then replace the `src/​aha.jl` file with
 +
 +```julia
 +module aha;
 +
 +using Random ## or anything else you wish to mixin
 +
 +export fun1
 +export var1
 +
 +fun1()= println("​fun 1 works"​)
 +fun2()= println("​fun 2 works"​)
 +
 +var1= 1
 +var2= 2
 +
 +end#module#
 +```
 +
 +you can then `Pkg.activate("​aha"​)` and the following will be true:
 +
 +```julia
 +julia> using Pkg;
 +
 +julia> Pkg.activate("​aha"​)
 +"/​Users/​ivo/​aha/​Project.toml"​
 +
 +julia> using aha ## may precompile the module aha
 +
 +julia> var1 ## was exported
 +1
 +
 +julia> var2 ## was not exported
 +ERROR: UndefVarError:​ var2 not defined
 +
 +julia> aha.var2 ##​ fully qualified works
 +2
 +
 +julia> var1= 2 ## module variables are read-only
 +ERROR: cannot assign variable aha.var1 from module Main
 +Stacktrace:
 +
 +julia> aha.var1= +2
 +ERROR: cannot assign variables in other modules
 +Stacktrace:
 +
 +julia> fun1() ##​ same thing for function. ​ the exported one
 +fun 1 works
 +
 +julia> fun2() ##​ was not exported
 +ERROR: UndefVarError:​ fun2 not defined
 +Stacktrace:
 +
 +julia> aha.fun2() ##​ fully qualified works
 +fun 2 works
 +
 +```
 +
 +
 +QUESTION FIXME : I am staring at the table in "​Summary of module usage",​ and I cannot see any difference between using and important, except that `using mymodule: x,p` does not allow extending x and p.  Correct?
 +
 +
 +
 +* [Modules](https://​docs.julialang.org/​en/​stable/​manual/​modules/​) are (global) namespaces. ​ `Main` is the global namespace. ​ `Core` and `Base` belong to Julia.
 +
 +* to export packages to all processes, `@everywhere using module` is perfectly fine.
 +
 +* there is a small distinction between `using module: x` and `import module: x` in whether x can be extended. ​ all other use cases of `using` and `import` are the same.
 +
 +* again, the import/​export/​write protection features only work for modules that are not loaded via `include()`,​ but as part of a package. ​ QUESTION FIXME: true?
 +
 +
 +
 +### Improving Module Startup Time
 +
 +Add `__precompile__()` the the top of the module file.  All included modules should also have `__precompile__()`
 +
 +
 +
 +
 +
 +
 +## Writing Tests
 +
 +### Inline Unit Tests
 +
 +Use the `@boundscheck` macro liberally inside functions with in-code tests. ​ Note that this will not reduce speed in production runs, because the compiler eliminates this together with other bounds checking.
 +
 +```juliarepl
 +julia> @boundscheck @assert( (true), "Your condition just failed. ​ Your program is buggy."​ )
 +```
 +
 +
 +### Poor Man's Inline Unit Tests
 +
 +A quick and dirty approach to small one-time programs is to have each function sit in its own file and have it be accompanied at its end by a set of tests. ​ Each Julia file contains at the end:
 +
 +```julia
 +
 +if ( (PROGRAM_FILE!=""​) && (!endswith(@__FILE__,​ PROGRAM_FILE)) )
 + @assert( f(0.2) == 1, "​f(0.2) does not yield 1, as it should, but $(f(0.2))"​ )
 +
 +        ## or even
 +        using Test
 +        @test( f(0.2) == 1 )
 +        ...
 +end#if
 +
 +```
 +
 +* When invoked from the command line, the julia tests are runs.
 +
 +* When included as part of a larger program, the test code is ignored.
 +
 +* To run the tests from the REPL, set PROGRAM_FILE equal to the name of the file
 +
 +
 +
 +
 +### Scaffolded (Rich Man's) External Unit Tests
 +
 +[Unit Testing](https://​docs.julialang.org/​en/​stable/​stdlib/​Test/​). ​ Typically, each file is accompanied by a test file of the same name, not in `src/` but in `test/`.
 +
 +
 +```juliarepl
 +julia> using Test
 +
 +julia> @test (true)
 +Test Passed
 +
 +julia> @test π ≈ 3.14 atol=0.01 ##​ note the atol preprocessor syntax.
 +Test Passed
 +
 +julia> @test_throws BoundsError [1, 2, 3][4]
 +Test Passed
 +      Thrown: BoundsError
 +
 +julia> @test_throws DomainError sqrt(-1)
 +Test Passed
 +      Thrown: DomainError
 +
 +julia> @test_throws ErrorException error("​hello"​)
 +Test Passed
 +      Thrown: ErrorException
 +
 +julia> @test (false)
 +Test Failed at none:1
 +  Expression: false
 +ERROR: There was an error during testing
 +
 +julia> @testset "My Tests" begin
 +           @test true
 +           @test (1 == 0)
 +           ​end;##​testset##​
 +My Tests: Test Failed at none:3
 +  Expression: 1 == 0
 +   ​Evaluated:​ 1 == 0
 +Stacktrace:
 + [1] macro expansion at ./none:3 [inlined]
 + [2] macro expansion at /​Users/​osx/​buildbot/​slave/​package_osx64/​build/​usr/​share/​julia/​stdlib/​v1.0/​Test/​src/​Test.jl:​1083 [inlined]
 + [3] top-level scope at ./none:2
 +Test Summary: | Pass  Fail  Total
 +My Tests      |    1     ​1 ​     2
 +
 +```
 +
 +
 +
 +## Speeding Up Production Code
 +
 +* Turn off boundschecking.
 +
 +QUESTION FIXME How do I turn it off?
 +
 +
 +
 +
 +
 +
 +## Debugging
 +
 +QUESTION FIXME (1) is Gallium the recommended debugger? ​ (2) is it now fixed?
 +
 +The debugger is called Gallium, but is not ready as of 0.6.2. ​ It is a work in progress. ​ Fortunately,​ it is usually a good substitute to add more print statements (or conditional stop-and-display statements) into the code itself. ​ One capability of the debugger is to display the source code of functions.
 +
 +```julianoeval
 +julia> using Gallium
 +
 +julia> @enter gcd(10,20)
 +In gcd(a, b) at intfuncs.jl:​31
 +31      a == 0 && return abs(b)
 +32      b == 0 && return abs(a)
 +33      za = trailing_zeros(a)
 +```
 +
 +Gallium offers the following commands:
 +
 +* Basic Commands:
 +
 +  - n steps to the next line
 +  - s steps into the next call
 +  - finish runs to the end of the function
 +  - bt shows a simple backtrace
 +  - `stuff` runs stuff in the current frame'​s context
 +  - fr v will show all variables in the current frame
 +  - f n where n is an integer, will go to the n-th frame.
 +
 +* Advanced commands:
 +
 +  - nc steps to the next call
 +  - se does one expression step
 +  - si does the same but steps into a call if a call is the next expression
 +  - sg steps into a generated function
 +
 +
 +
 +
 +### StackFrame Handling
 +
 +FIXME explain more using StackFrames;​ StackFrame .  In [StackTraces](https://​docs.julialang.org/​en/​stable/​stdlib/​stacktraces/​).
 +
 +QUESTION FIXME Can I turn off stackframe printing that is *not* in my jl file?  Or completely turn it off?
 +
 +
 +
 +#### Suppressing Stack Traces
 +
 +
 +#### Also TraceCalls.jl
 +
 +TraceCalls is broken as of Julia 1.0.  It could nicely tell you what the sequence was of what was called.
 +
 +```julianoeval
 +julia> using Optim, TraceCalls
 +
 +julia> @traceable logplus10(x) = log(x[1]+10)
 +logplus10 (generic function with 1 method)
 +
 +julia> TraceCalls.store(v::​Vector) = copy(v)
 +
 +julia> strace = @stacktrace (Optim, Calculus) optimize(logplus10,​ [0.0], BFGS())
 +
 +```
 +
 +
 +
 +
 +
 +## Profiling
 +
 +TODO check into http://​docs.junolab.org/​latest/​man/​juno_frontend.html#​Profiler-1 21
 +
 +
 +```julianoeval
 +julia> using Profile, Statistics, Random
 +
 +julia> Random.seed!(0);​
 +
 +julia> function testme()
 +                A= rand(200,​200,​400)
 +                B= A.^2
 +                C = mean(A)+mean(B)
 +                D = C^2
 +              end;##​function testme##
 +
 +julia> testme()
 +0.6943385758607669
 +
 +julia> @profile testme()
 +0.6940778198310061
 +
 +julia> Profile.print(format=:​flat,​ mincount=60)
 + Count File                                                                                                    Line Function
 +    61 ./​boot.jl ​                                                                                               398 Type
 +    61 ./​boot.jl ​                                                                                               405 Type
 +    61 ./​boot.jl ​                                                                                               411 Type
 +   142 ./​boot.jl ​                                                                                               319 eval(::​Module,​ ::Any)
 +    79 ./​none ​                                                                                                    2 testme()
 +   210 ./​reduce.jl ​                                                                                             162 mapreduce_impl(::​typeof(identity),​ ::​typeof(Base.a...
 +   100 ./​simdloop.jl ​                                                                                            67 macro expansion
 +    96 ./​simdloop.jl ​                                                                                            68 macro expansion
 +   142 ./​task.jl ​                                                                                               259 (::​getfield(REPL,​ Symbol("##​28#​29"​)){REPL.REPLBack...
 +   142 /​Users/​osx/​buildbot/​slave/​package_osx64/​build/​usr/​share/​julia/​stdlib/​v1.0/​Profile/​src/​Profile.jl ​         25 top-level scope
 +   142 /​Users/​osx/​buildbot/​slave/​package_osx64/​build/​usr/​share/​julia/​stdlib/​v1.0/​REPL/​src/​REPL.jl ​               85 eval_user_input(::​Any,​ ::​REPL.REPLBackend)
 +   142 /​Users/​osx/​buildbot/​slave/​package_osx64/​build/​usr/​share/​julia/​stdlib/​v1.0/​REPL/​src/​REPL.jl ​              117 macro expansion
 +    79 /​Users/​osx/​buildbot/​slave/​package_osx64/​build/​usr/​share/​julia/​stdlib/​v1.0/​Random/​src/​Random.jl ​          243 rand
 +    79 /​Users/​osx/​buildbot/​slave/​package_osx64/​build/​usr/​share/​julia/​stdlib/​v1.0/​Random/​src/​Random.jl ​          257 rand
 +    79 /​Users/​osx/​buildbot/​slave/​package_osx64/​build/​usr/​share/​julia/​stdlib/​v1.0/​Random/​src/​Random.jl ​          258 rand
 +
 +```
 +
 +Unfortunately,​ even the :flat output is not user-friendly. ​ We would really like to see the time spent in *our* four statements, but even the flat version here is not at all clear. ​ The default nonflat output, which is really what we often want, is even harder to digest.
 +
 +For now, try to extract from this that the REPL lines contain the four statements. ​ Unfortunately,​ there is no clear indication which statement refers to which count.
 +
 +
 +
 +## Benchmarking
 +
 +Julia is a compiled language. ​ It is silly to benchmark small problems and compare the performance to interpreted languages. ​ Julia only provides speed advantages for problems where run-time >> compile-time.
 +
 +The `@time` macro is not quick and convenient, but not reliable.
 +
 +### Basic BenchmarkTools
 +
 +Use the `@btime` macro from package `BenchmarkTools`,​ and wrap the code to be benchmarked in a `begin...end` block:
 +
 +```julianoeval
 +using BenchmarkTools;​
 +
 +julia> @btime begin; rand(100_000);​ end;#end##
 +  67.220 μs (2 allocations:​ 781.33 KiB)
 +```
 +
 +
 +### Quoting Functions and Variables For BenchmarkTools
 +
 +When you want to benchmark a passed function, you may need to precede it with a '​$'​ sign for interpolation,​ e.g., `function testme(afun);​ @btime $afun(2); end`.  Similarly
 +
 +```julianoeval
 +julia> for i=2:9; println(i); @btime ( n=2^$i; sum(qr( reshape( collect( 1:(n*n)), (n,n) ) ) )); end
 +```
 +
 +This becomes more complex with symbols, because they are themselves used.  Here is an example:
 +
 +```julianoeval
 +julia> using BenchmarkTools
 +
 +julia> wfunprint= :sqrt; wfunnoprint= :exp;
 +
 +julia> pfun( fw::Symbol )= ((fw == :sqrt) && println("​p:​ $fw bye"​));​
 +
 +julia> pfun(wfunprint); ​                       ## unquoted print
 +p: sqrt bye
 +
 +julia> @btime pfun($(QuoteNode(wfunnoprint)));​ ## benchmark print needs quote of symbol
 +  0.023 ns (0 allocations:​ 0 bytes)
 +
 +julia> function mybmk( wfunpassedin::​Symbol )
 +           qfun( fw::Symbol )= ((fw == :sqrt) && println("​q:​ $fw bye"))
 +           ​qfun(wfunprint); ​           ## ok
 +           # pfun does not need quoting, because it is global
 +           ​@btime pfun($(QuoteNode(wfunpassedin)));​
 +           # qfun needs quoting, and wfunpassedin needs special quoting
 +           ​@btime $qfun($(QuoteNode(wfunpassedin)));​
 +       ​end;##​function##​
 +
 +julia> mybmk( wfunnoprint );
 +q: sqrt bye
 +  0.023 ns (0 allocations:​ 0 bytes)
 +  0.024 ns (0 allocations:​ 0 bytes)
 +
 +```
 +
 +See also [Quoting](https://​stackoverflow.com/​questions/​41089019/​disambiguate-the-various-quoting-mechanisms-in-julia-metaprogramming)
 +
 +
 +
 +### Storing benchmarked output for later processing
 +
 +```julianoeval
 +julia> t= @benchmark( eig(rand(20,​20)) )
 +
 +julia> dump(t)
 +
 +julia> median(t.times),​ t.memory()
 +
 +```
 +
 +
 +
 +## Timer Outputs
 +
 +Timer Outputs could also be useful to find where in a function most time is spend. Although it is not that handy to use, the information helps to verify impressions from the profiling.
 +
 +```julianoeval
 +julia> using TimerOutputs
 +
 +julia> t= TimerOutput();​
 +
 +julia> function testme()
 +               ​@timeit t "​rand"​ A= rand(200,​200,​400)
 +               ​@timeit t "​element square"​ B= A.^2
 +               ​@timeit t "​mean"​ C = mean(A)+mean(B)
 +               ​@timeit t "​square mean" D = C^2
 +       ​end;#​function testme##
 +
 +julia> testme();
 +
 +julia> t
 + ​─────────────────────────────────────────────────────────────────────────
 +                                  Time                   ​Allocations
 +                          ────────────────────── ​  ​───────────────────────
 +     Tot / % measured: ​        26.7s / 0.94%            395MiB / 62.0%
 +
 + ​Section ​         ncalls ​    ​time ​  ​%tot ​    ​avg ​    ​alloc ​  ​%tot ​     avg
 + ​─────────────────────────────────────────────────────────────────────────
 + ​element square ​       1    132ms  52.7%   ​132ms ​   122MiB ​ 49.8%   ​122MiB
 + ​rand ​                 1   ​93.3ms ​ 37.3%  93.3ms ​   123MiB ​ 50.2%   ​123MiB
 + ​mean ​                 1   ​24.9ms ​ 10.0%  24.9ms ​        ​- ​ 0.00%        -
 + ​square mean           ​1 ​   845ns  0.00%   ​845ns ​        ​- ​ 0.00%        -
 + ​─────────────────────────────────────────────────────────────────────────
 +```
 +
 +
 +
 +
 +# Backmatter
 +
 +## Useful Packages on Julia Repository
 +
 +* [Traceur = Analyze code for Julia Performance Tips](https://​github.com/​MikeInnes/​Traceur.jl)
 +
 +## Notes
 +
 +### Tasks and RemoteChannels
 +
 +QUESTION FIXME tasks: What are tasks for, and how do I use them?  Give simplest example. (Andreas'​ comment: maybe reconsider if Tasks and Channels belong in Part I. These topics are pretty advanced and many users can do fine without knowing about them. Those two are interested might be better off by just reading the manual entries.)
 +
 +### RemoteChannels
 +
 +QUESTION FIXME remotechannels:​ What are they for, and how do I use them?  Give simplest example.
 +
 +```julianoeval
 +using Distributed
 +addprocs(2)
 +const channel= RemoteChannel(()->​Channel{Int}(1))
 +
 +@async pmap(1:20) do i
 +    isready(channel) && println(i)
 +    sleep(3)
 +end
 +```
 +
 +
 +
 +## References
 +
 +- http://​docs.julialang.org/​en/​release-0.5/​manual/​noteworthy-differences/​
 +- https://​gist.github.com/​cuckookernel/​9784889
 +
 +
 +
  
programming.txt · Last modified: 2019/01/02 11:39 (external edit)