User Tools

Site Tools


programming
snippet.juliarepl
julia> pkgchk( [ "julia" => v"1.0.2", "TimerOutputs" => v"0.4.0" ] )

Advanced Programming

See also functions.

  • Benchmarking and Unit-Testing work reasonably well.
  • Debugging and profiling are rudimentary in 0.6.2.

Coding Style

Julia Official

The Julia developers request the code style to be

  • lowercase variable, function, and macro names,
  • capitalize Types and Modules,
  • avoid underscores (CamelCase preferred).

The full style guide is here.

Cookbook Inofficial

I request that code be written to do the following

  • make all variables const as soon as possible
  • explicitly (strongly) type all function inputs and outputs
  • use the @boundscheck macro liberally inside functions with in-code tests.

Programming Projects With Scaffolding

The manual describes the process in more detail.

Generation

snippet.julianoeval
[download only julia statements]
julia> using Pkg
 
julia> Pkg.generate("MyFirstPackage")
Generating project MyFirstProject:
    MyFirstProject/Project.toml
    MyFirstProject/src/MyFirstProject.jl

The MyFirstPackage.toml file is

snippet.julia
authors = ["ivo welch .welch@gmail.com>"]
name = "MyFirstProject"
uuid = "73461ff8-ab02-11e8-1d84-2180feccd51a"
version = "0.1.0"

[deps]

There are elaborate ways to declare dependencies to other packages, all explained in the manual.

Modules

The src/MyFirstProject.jl file is a module in a jl file. The default is

snippet.julia
module MyFirstProject

greet() = print("Hello World!")

end # module

Please see the extended description below.

Running

snippet.julianoeval
[download only julia statements]
julia> Pkg.activate("./MyFirstProject")		## where to find the project.
"/Users/ivo/MyFirstProject/Project.toml"
 
julia> import MyFirstProject			## load the project.  using = import
[ Info: Precompiling MyFirstProject [73461ff8-ab02-11e8-1d84-2180feccd51a]
 
julia> MyFirstProject.greet()			## run the project
Hello World!

Add tests to the project

snippet.julianoeval
[download only julia statements]
julia> print( open("MyFirstProject/test/runtests.jl", "w"), "println(\"Testing...\")" ); gc();
 
julia> Pkg.test("MyFirstProject")
Testing...
   Testing MyFirstProject tests passed

Changing the LOAD_PATH

When a package is included, julia automatically knows that other included .jl files can also be in the package src directory. this makes including them seamless.

Nevertheless, it is sometimes useful to add to the loadpath, so that related packages and files can be found. This can be done with push!(LOAD_PATH, "./").

Install Your Package For Your Own Use

  • Packages for one project can be kept in the local directory, which is activated with Pkg.activate(".").
  • The default location for user packages that should be available systemwide (i.e. from everywhere and without a mandatory Pkg.activate) is FIXME.

Github Storage

Docs, Chapter 35.

snippet.sh
[download only julia statements]
$ git config --global user.name "Me Is"
$ git config --global user.email "email@gmail.com"

QUESTION FIXME Any Special Support for Git? Or do we just use the standard CLI from the Unix shell?

Contribute Your Package to the Repository

FIXME PkgDev is broken in Julia 1.0 as of Sep 2018

Also, PkgDev.register("FooBar") and PkgDev.publish().

Extended Discussion of Modules For Packages

Modules (namespaces) are really designed for use in packages. They have import/export and variable protection facilities, but when used as part of an ordinary program (i.e., just included with *.jl), they loose all these features. Moreover, in this case, you can only access their variables in fully qualified form (i.e., modulename.variablename).

If you Pkg.generate('aha') and then replace the src/aha.jl file with

snippet.julia
module aha;

using Random	## or anything else you wish to mixin

export fun1
export var1

fun1()= println("fun 1 works")
fun2()= println("fun 2 works")

var1= 1
var2= 2

end#module#

you can then Pkg.activate("aha") and the following will be true:

snippet.julia
julia> using Pkg;

julia> Pkg.activate("aha")
"/Users/ivo/aha/Project.toml"

julia> using aha			## may precompile the module aha

julia> var1				## was exported
1

julia> var2				## was not exported
ERROR: UndefVarError: var2 not defined

julia> aha.var2				## fully qualified works
2

julia> var1= 2				## module variables are read-only
ERROR: cannot assign variable aha.var1 from module Main
Stacktrace:

julia> aha.var1= +2
ERROR: cannot assign variables in other modules
Stacktrace:

julia> fun1()				## same thing for function.  the exported one
fun 1 works

julia> fun2()				## was not exported
ERROR: UndefVarError: fun2 not defined
Stacktrace:

julia> aha.fun2()			## fully qualified works
fun 2 works

QUESTION FIXME : I am staring at the table in “Summary of module usage”, and I cannot see any difference between using and important, except that using mymodule: x,p does not allow extending x and p. Correct?

  • Modules are (global) namespaces. Main is the global namespace. Core and Base belong to Julia.
  • to export packages to all processes, @everywhere using module is perfectly fine.
  • there is a small distinction between using module: x and import module: x in whether x can be extended. all other use cases of using and import are the same.
  • again, the import/export/write protection features only work for modules that are not loaded via include(), but as part of a package. QUESTION FIXME: true?

Improving Module Startup Time

Add __precompile__() the the top of the module file. All included modules should also have __precompile__()

Writing Tests

Inline Unit Tests

Use the @boundscheck macro liberally inside functions with in-code tests. Note that this will not reduce speed in production runs, because the compiler eliminates this together with other bounds checking.

snippet.juliarepl
julia> @boundscheck @assert( (true), "Your condition just failed.  Your program is buggy." )

Poor Man's Inline Unit Tests

A quick and dirty approach to small one-time programs is to have each function sit in its own file and have it be accompanied at its end by a set of tests. Each Julia file contains at the end:

snippet.julia
if ( (PROGRAM_FILE!="") && (!endswith(@__FILE__, PROGRAM_FILE)) )
	@assert( f(0.2) == 1, "f(0.2) does not yield 1, as it should, but $(f(0.2))" )

        ## or even
        using Test
        @test( f(0.2) == 1 )
        ...
end#if
  • When invoked from the command line, the julia tests are runs.
  • When included as part of a larger program, the test code is ignored.
  • To run the tests from the REPL, set PROGRAM_FILE equal to the name of the file

Scaffolded (Rich Man's) External Unit Tests

Unit Testing. Typically, each file is accompanied by a test file of the same name, not in src/ but in test/.

snippet.juliarepl
julia> using Test

julia> @test (true)
Test Passed

julia> @test π ≈ 3.14 atol=0.01		## note the atol preprocessor syntax.
Test Passed

julia> @test_throws BoundsError [1, 2, 3][4]
Test Passed
      Thrown: BoundsError

julia> @test_throws DomainError sqrt(1)
Test Passed
      Thrown: DomainError

julia> @test_throws ErrorException error("hello")
Test Passed
      Thrown: ErrorException

julia> @test (false)
Test Failed at none:1
  Expression: false
ERROR: There was an error during testing

julia> @testset "My Tests" begin
           @test true
           @test (1 == 0)
           end;##testset##
My Tests: Test Failed at none:3
  Expression: 1 == 0
   Evaluated: 1 == 0
Stacktrace:
 [1] macro expansion at ./none:3 [inlined]
 [2] macro expansion at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/Test/src/Test.jl:1083 [inlined]
 [3] top-level scope at ./none:2
Test Summary: | Pass  Fail  Total
My Tests      |    1     1      2

Speeding Up Production Code

  • Turn off boundschecking.

QUESTION FIXME How do I turn it off?

Debugging

QUESTION FIXME (1) is Gallium the recommended debugger? (2) is it now fixed?

The debugger is called Gallium, but is not ready as of 0.6.2. It is a work in progress. Fortunately, it is usually a good substitute to add more print statements (or conditional stop-and-display statements) into the code itself. One capability of the debugger is to display the source code of functions.

snippet.julianoeval
[download only julia statements]
julia> using Gallium
 
julia> @enter gcd(10,20)
In gcd(a, b) at intfuncs.jl:31
31      a == 0 && return abs(b)
32      b == 0 && return abs(a)
33      za = trailing_zeros(a)

Gallium offers the following commands:

  • Basic Commands:
    • n steps to the next line
    • s steps into the next call
    • finish runs to the end of the function
    • bt shows a simple backtrace
    • stuff runs stuff in the current frame's context
    • fr v will show all variables in the current frame
    • f n where n is an integer, will go to the n-th frame.
  • Advanced commands:
    • nc steps to the next call
    • se does one expression step
    • si does the same but steps into a call if a call is the next expression
    • sg steps into a generated function

StackFrame Handling

FIXME explain more using StackFrames; StackFrame . In StackTraces.

QUESTION FIXME Can I turn off stackframe printing that is not in my jl file? Or completely turn it off?

Suppressing Stack Traces

Also TraceCalls.jl

TraceCalls is broken as of Julia 1.0. It could nicely tell you what the sequence was of what was called.

snippet.julianoeval
[download only julia statements]
julia> using Optim, TraceCalls
 
julia> @traceable logplus10(x) = log(x[1]+10)
logplus10 (generic function with 1 method)
 
julia> TraceCalls.store(v::Vector) = copy(v)
 
julia> strace = @stacktrace (Optim, Calculus) optimize(logplus10, [0.0], BFGS())

Profiling

TODO check into http://docs.junolab.org/latest/man/juno_frontend.html#Profiler-1 21

snippet.julianoeval
[download only julia statements]
julia> using Profile, Statistics, Random
 
julia> Random.seed!(0);
 
julia> function testme()
                A= rand(200,200,400)
                B= A.^2
                C = mean(A)+mean(B)
                D = C^2
              end;##function testme##
 
julia> testme()
0.6943385758607669
 
julia> @profile testme()
0.6940778198310061
 
julia> Profile.print(format=:flat, mincount=60)
 Count File                                                                                                    Line Function
    61 ./boot.jl                                                                                                398 Type
    61 ./boot.jl                                                                                                405 Type
    61 ./boot.jl                                                                                                411 Type
   142 ./boot.jl                                                                                                319 eval(::Module, ::Any)
    79 ./none                                                                                                     2 testme()
   210 ./reduce.jl                                                                                              162 mapreduce_impl(::typeof(identity), ::typeof(Base.a...
   100 ./simdloop.jl                                                                                             67 macro expansion
    96 ./simdloop.jl                                                                                             68 macro expansion
   142 ./task.jl                                                                                                259 (::getfield(REPL, Symbol("##28#29")){REPL.REPLBack...
   142 /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/Profile/src/Profile.jl          25 top-level scope
   142 /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/REPL/src/REPL.jl                85 eval_user_input(::Any, ::REPL.REPLBackend)
   142 /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/REPL/src/REPL.jl               117 macro expansion
    79 /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/Random/src/Random.jl           243 rand
    79 /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/Random/src/Random.jl           257 rand
    79 /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/Random/src/Random.jl           258 rand

Unfortunately, even the :flat output is not user-friendly. We would really like to see the time spent in our four statements, but even the flat version here is not at all clear. The default nonflat output, which is really what we often want, is even harder to digest.

For now, try to extract from this that the REPL lines contain the four statements. Unfortunately, there is no clear indication which statement refers to which count.

Benchmarking

Julia is a compiled language. It is silly to benchmark small problems and compare the performance to interpreted languages. Julia only provides speed advantages for problems where run-time » compile-time.

The @time macro is not quick and convenient, but not reliable.

Basic BenchmarkTools

Use the @btime macro from package BenchmarkTools, and wrap the code to be benchmarked in a begin...end block:

snippet.julianoeval
[download only julia statements]
using BenchmarkTools;
 
julia> @btime begin; rand(100_000); end;#end##
  67.220 μs (2 allocations: 781.33 KiB)

Quoting Functions and Variables For BenchmarkTools

When you want to benchmark a passed function, you may need to precede it with a '$' sign for interpolation, e.g., function testme(afun); @btime $afun(2); end. Similarly

snippet.julianoeval
[download only julia statements]
julia> for i=2:9; println(i); @btime ( n=2^$i; sum(qr( reshape( collect( 1:(n*n)), (n,n) ) ) )); end

This becomes more complex with symbols, because they are themselves used. Here is an example:

snippet.julianoeval
[download only julia statements]
julia> using BenchmarkTools
 
julia> wfunprint= :sqrt; wfunnoprint= :exp;
 
julia> pfun( fw::Symbol )= ((fw == :sqrt) && println("p: $fw bye"));
 
julia> pfun(wfunprint);                        ## unquoted print
p: sqrt bye
 
julia> @btime pfun($(QuoteNode(wfunnoprint))); ## benchmark print needs quote of symbol
  0.023 ns (0 allocations: 0 bytes)
 
julia> function mybmk( wfunpassedin::Symbol )
           qfun( fw::Symbol )= ((fw == :sqrt) && println("q: $fw bye"))
           qfun(wfunprint);            ## ok
           # pfun does not need quoting, because it is global
           @btime pfun($(QuoteNode(wfunpassedin)));
           # qfun needs quoting, and wfunpassedin needs special quoting
           @btime $qfun($(QuoteNode(wfunpassedin)));
       end;##function##
 
julia> mybmk( wfunnoprint );
q: sqrt bye
  0.023 ns (0 allocations: 0 bytes)
  0.024 ns (0 allocations: 0 bytes)

See also Quoting

Storing benchmarked output for later processing

snippet.julianoeval
[download only julia statements]
julia> t= @benchmark( eig(rand(20,20)) )
 
julia> dump(t)
 
julia> median(t.times), t.memory()

Timer Outputs

Timer Outputs could also be useful to find where in a function most time is spend. Although it is not that handy to use, the information helps to verify impressions from the profiling.

snippet.julianoeval
[download only julia statements]
julia> using TimerOutputs
 
julia> t= TimerOutput();
 
julia> function testme()
               @timeit t "rand" A= rand(200,200,400)
               @timeit t "element square" B= A.^2
               @timeit t "mean" C = mean(A)+mean(B)
               @timeit t "square mean" D = C^2
       end;#function testme##
 
julia> testme();
 
julia> t
 ─────────────────────────────────────────────────────────────────────────
                                  Time                   Allocations
                          ──────────────────────   ───────────────────────
     Tot / % measured:         26.7s / 0.94%            395MiB / 62.0%
 
 Section          ncalls     time   %tot     avg     alloc   %tot      avg
 ─────────────────────────────────────────────────────────────────────────
 element square        1    132ms  52.7%   132ms    122MiB  49.8%   122MiB
 rand                  1   93.3ms  37.3%  93.3ms    123MiB  50.2%   123MiB
 mean                  1   24.9ms  10.0%  24.9ms         -  0.00%        -
 square mean           1    845ns  0.00%   845ns         -  0.00%        -
 ─────────────────────────────────────────────────────────────────────────

Backmatter

Useful Packages on Julia Repository

Notes

Tasks and RemoteChannels

QUESTION FIXME tasks: What are tasks for, and how do I use them? Give simplest example. (Andreas' comment: maybe reconsider if Tasks and Channels belong in Part I. These topics are pretty advanced and many users can do fine without knowing about them. Those two are interested might be better off by just reading the manual entries.)

RemoteChannels

QUESTION FIXME remotechannels: What are they for, and how do I use them? Give simplest example.

snippet.julianoeval
[download only julia statements]
using Distributed
addprocs(2)
const channel= RemoteChannel(()->Channel{Int}(1))
 
@async pmap(1:20) do i
    isready(channel) && println(i)
    sleep(3)
end

References

programming.txt · Last modified: 2018/11/22 20:48 (external edit)