User Tools

Site Tools


start

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
start [2018/06/13 15:08]
127.0.0.1 external edit
start [2018/12/28 10:27] (current)
Line 1: Line 1:
  
-~~CLOSETOC~~+ ​**This book is copyrighted,​ but free.**
  
-~~TOC 1-3 wide~~+ The author (Ivo Welch) grants permission to use individual snippets freely in other code or in postings (preferably with attribution). ​ But please do not copy chapters wholesale.
  
-# Julia Cookbook (0.7.0-alpha)+ 
 + 
 +# Julia 1.0 Cookbook
  
 [comment]: <> (This is a comment, it will not be included) [comment]: <> (This is a comment, it will not be included)
Line 12: Line 14:
  
  
-### About This Cookbook+0. [[preface|Preface]] --- about this cookbook, about Julia, about Julia computing, other recommended reading, license
  
-* The target audience for this cookbook are novice and intermediate ​Julia users with some prior programming experience (e.g., in R, perl, python, C, Fortran), and especially researchers with an interest in the social sciences.+## Part 1: Julia Core
  
-This cookbook ​is for finding quick solutions ​of common patterns and questions, even for regular Julia programmers.+ WARNING Part 1 is approaching first-draft status. ​ Many of the chapters below are nearly done.  The number of CHECKMARK indicates the progress. ​ These checkmarks are updated irregularly.
  
-* This cookbook ​is **not** well-suited ​to learning programming.+Julia consists of a core computer language and packages. ​ Some packages come with Julia. ​ Others including very important basic packages need to be installed (which is easy). ​ Packages can add essential functionality (e.g., DataFrames as dataset storage), programming tools (e.g., BenchmarkTools or Parallel Programming support), and/or entire applications (e.g., statistical analysis or plotting). ​ Julia is *not* a lightweight language, with few principal features to remember---like C or lisp.  It is a beast with many heads and appendages, tremendously powerful, but also easy to get lost in---similar to R (with CRAN), python (with PyPI), and perl (with CPAN) in this respect.
  
-* This cookbook is **not** well-suited for experienced julia programmers and developers, or those who want to read a book to become one. 
  
-The exposition methods are minimalistic examples, which should speak for themselves.  ​The reader should immediately understand the functionality by looking at the (working) example. ​ Long additional text explanations are (sometimes necessary) shortcomings.+1.  ​Introduction
  
 +    1. [[installation|Installation]] also REPL (Interactive) and Batch Use. CHECKMARK CHECKMARK CHECKCROSS 6
  
-### About Julia+    1. [[packages|Packages]],​ installing, removing, using. ​ CHECKMARK CHECKMARK CHECKMARK 1
  
-Julia is a next-generation programming language with many innovations From my perspective,​ the most noteworthy general aspects ​about Julia are:+    1[[inquiring|Inquiring]] ​about object types and structures, methods and functions, naming conventions,​ etc.  CHECKMARK CHECKMARK CHECKMARK 2
  
-* Julia arrays (vectors and matrices, easy to use for both Int and Float forms) are first-class design features, making Julia a great language for scientific computing. 
  
-* Julia is strongly typed, making it easier both to find bugs earlier and to write efficient programs.+2 ​Control Flow
  
-* Julia relies greatly on function overloading,​ where a function with the same name can dispatch based on argument type.+    1[[controlflow|Conditionals (Bools). Exceptions. Operators. Notes]] ​ CHECKMARK CHECKMARK CHECKMARK 1
  
-* Julia is compiled, with requisite compilation overhead. ​ (Think of Julia more as C++, not as perl or R.)  It is slow for short programs (compared to python and R), but it is (often blazingly) fast for long programs. 
  
- +3. Scalars
- +
-I am recommending Julia for programming,​ because: +
- +
-* Julia user code is cleaner and more logical than python or perl code, where (float) matrices are subsequent grafts. ​ It is cleaner than R, which has never managed to decide whether it is primarily an interactive environment or a programming language. ​ R's weak typing means bugs often manifest themselves very late, and debugging R can be akin to painful analysis of mystical incantations. +
- +
-* Julia is less bogged down by its own history than R, Python, or perl.  The developers seem to get along and often help users (incl. early users like myself). ​ The julia system has, as of 2018, not yet calcified to the point where even good and simple changes cause endless controversies. ​ Every language must calcify sooner or later, but Julia will do so with the experiences of 2020, not with those of 1990. +
- +
-* Think of Julia as a "​heavy-duty"​ tool and R as a "​light-duty"​ tool.  It would be silly to cut a hedge with scissors, or to cut paper with a hedge trimmer. ​ Both can serve to do the other job in a pinch, but they really serve rather different purposes. +
- +
- +
-In my mind, Julia makes for a worse interactive environment than R (due to R's sloppy language and instant interpreted response speed), and a better programming environment than R (due to Julia'​s more careful and sane syntax and faster program run speed). ​ Let me state this for the third time. +
- +
-* WARNING Julia is often an order of magnitude faster than R/​python/​perl for long programs. ​ But Julia is also an order of magnitude slower than R/​python/​perl for short scripts and interactive use.   +
- +
- +
- +
-### About JuliaComputing and JuliaPro +
- +
-[Julia](https://​en.wikipedia.org/​wiki/​Julia_(programming_language) is an open-source next-generation scientific computer language, designed by some amazing people at JuliaComputing (such as Jeff Bezanson, Alan Edelman, Stefan Karpinski, Viral B. Shah, and others). ​ It has enthusiastic supporters that have answered many of my questions while I learned Julia (and while I wrote this cookbook). ​ My own contribution for this cookbook was curating, i.e., keeping track of and organizing the smallest clear examples I could think of while I was learning Julia myself. ​ The [Perl Cookbook](http://​docstore.mik.ua/​orelly/​perl/​cookbook/​ch04_01.htm) was my primary inspiration for this book.  The perl cookbook is what made perl usable for me when I learned perl.  (Thank you, Tom and Nathan!) ​ I hope this Julia cookbook will help making Julia a success with new users. +
- +
- +
-[JuliaComputing](https://​juliacomputing.com/​) employs the core developers for Julia. ​ To help finance Julia'​s development,​ please support them by purchasing their products: +
- +
-1. [JuliaPro](https://​juliacomputing.com/​products/​juliapro) is Julia with a set of curated and more stable packages, an IDE, and enterprise support. +
- +
-2. [JuliaBox](https://​juliacomputing.com/​products/​juliabox) is a web-based interface to a working and curated Julia, sold for a modest per month subscription price. +
- +
- +
-JuliaComputer further sells, but I have not yet used the following:​ +
- +
-2. [JuliaBox](https://​juliacomputing.com/​products/​juliabox) is a web-based interface to a working and curated Julia, sold for a modest per month subscription price. +
- +
-3. [JuliaFin](https://​juliacomputing.com/​products/​juliafin) +
- +
-4. [JuliaRun](https://​juliacomputing.com/​products/​juliarun) is for large distributed cluster operations. +
- +
-5. [JuliaDB](https://​juliacomputing.com/​products/​juliadb) is for operations on large disk-based data sets. +
- +
- +
-# Julia Core Language +
- +
- +
-Julia consists of a core computer language and packages. ​ Packages can add essential functionality (e.g., Missing values or DataFrame as dataset storage), programming tools (e.g., BenchmarkTools or Parallel Programming support), and/or entire applications (e.g., statistical analysis or plotting). ​ Julia is *not* a lightweight language, with few principal features to remember, like C or lisp.  It is a beast with many heads and appendages, tremendously powerful, but also easy to get lost in---similar to R (with CRAN), python (with PyPI), and perl (with CPAN) in this respect. +
- +
-* WARNING Some of the chapters below are nearly done.  Others are just scrapbooks. ​ The number of CHECKMARK indicates the progress. ​ These checkmarks are updated irregularly. +
- +
- +
-0. Introduction +
- +
-   1. [[installation|Installation and Packages]] also REPL (Interactive) and Batch Use. CHECKMARK CHECKMARK +
- +
-   2. [[inquiring|Inquiring]] about object types and structures, methods and functions, naming conventions,​ etc.  CHECKMARK CHECKMARK CHECKMARK +
- +
-   -- +
- +
- +
-1. Command Flow +
- +
-   1. [[commandflow|Conditionals (Bools). Exceptions. Operators. Notes]] ​ CHECKMARK CHECKMARK CHECKMARK +
- +
-   -- +
- +
-1. Scalars+
  
    1. [[numbers|Numeric Scalars and Types]] CHECKMARK CHECKMARK CHECKMARK    1. [[numbers|Numeric Scalars and Types]] CHECKMARK CHECKMARK CHECKMARK
Line 106: Line 43:
    1. [[datetime|Date and Time]] CHECKMARK CHECKMARK CHECKMARK    1. [[datetime|Date and Time]] CHECKMARK CHECKMARK CHECKMARK
  
-   -- 
  
-1. Strings+4. Strings 
    1. [[strings|Basics]]:​    1. [[strings|Basics]]:​
-   ​concatenations,​ substrings, conversions,​ padding, capitalization,​ characterizations,​ parsing, escaping, etc.  CHECKMARK CHECKMARK CHECKMARK+   ​concatenations,​ substrings, conversions,​ padding, capitalization,​ characterizations,​ parsing, escaping, etc.  CHECKMARK CHECKMARK CHECKMARK ​1
  
    1. [[regex|Regular Expressions]]: ​ finding and matching complex patterns. ​ CHECKMARK CHECKMARK CHECKMARK    1. [[regex|Regular Expressions]]: ​ finding and matching complex patterns. ​ CHECKMARK CHECKMARK CHECKMARK
  
-   -- 
  
 +5. [[missings|Missing Values and Types]] ​ CHECKMARK CHECKMARK CHECKMARK
  
-1. Arrays and Tuples+6. Arrays and Tuples
  
-   1. [[arraysintro|Introduction]] ​ CHECKMARK CHECKMARK CHECKMARK+   1. [[arraysintro|Introduction]] ​ CHECKMARK CHECKMARK CHECKMARK ​1
  
-   1. [[arraysvector|Vectors]] ​ CHECKMARK CHECKMARK CHECKMARK+   2. [[arraysvector|Vectors]] ​ CHECKMARK CHECKMARK CHECKMARK ​1
  
-   1. [[arraysortandsets|Sorting and Sets]] ​ CHECKMARK CHECKMARK CHECKMARK+   3. [[arraysortandsets|Sorting and Sets]] ​ CHECKMARK CHECKMARK CHECKMARK
  
-   1. [[arraysmatrix|Matrices]] ​ CHECKMARK CHECKMARK CHECKMARK+   4. [[arraysmatrix|Matrices]] ​ CHECKMARK CHECKMARK CHECKMARK
  
-   -- 
  
 +7. Associative Arrays (Key-Value)
  
-1. [[missings|Missing Values ​and Types]]  CHECKMARK CHECKMARK CHECKMARK+   1. [[dicts|Dictionaries (and Pairs)]]: (=maps=hashes),​ could be viewed as run-time structs. ​  CHECKMARK CHECKMARK CHECKMARK ​1
  
-   -- +   2. [[structs|Structures]]:​ could be viewed as compile-time dictionaries. ​  ​CHECKMARK CHECKMARK CHECKMARK
- +
-1. Associative Arrays (Key-Value) +
- +
-   1. [[dicts|Dictionaries]]:​ (=maps=hashes),​ could be viewed as run-time structs. ​  ​CHECKMARK CHECKMARK CHECKMARK +
- +
-   1. [[structs|Structures]]:​ could be viewed as compile-time dictionaries. ​  ​CHECKMARK CHECKMARK CHECKMARK +
- +
-   -- +
- +
-1. Programming and Functions +
- +
-   1. [[functions|Defining and calling functions]] ​ CHECKMARK CHECKMARK CHECKMARK +
- +
-   1. [[parallel|Parallel Processing]]: ​ Basics and Embarrassingly Parallel Processing CHECKMARK +
- +
-   1. [[programming|Programming Tools]]: Debugging, Profiling, Benchmarking,​ Unit-Testing,​ Writing Packages. +
- +
-   --+
  
  
 1. DataFrames 1. DataFrames
  
-   1. [[dataframeintro|Introduction]]:​ building and accessing dataframes ​ CHECKMARK CHECKMARK CHECKMARK+   1. [[dataframeintro|Introduction]]:​ building and accessing dataframes ​ CHECKMARK CHECKMARK CHECKMARK ​1
  
    1. [[dataframemissing|Missing and NaN]]: in the context of data frames ​ CHECKMARK CHECKMARK CHECKMARK    1. [[dataframemissing|Missing and NaN]]: in the context of data frames ​ CHECKMARK CHECKMARK CHECKMARK
Line 165: Line 84:
    1. [[dataframecomplex|Complex Operations]]:​ sorts, joins, by-group operations, wide-narrow conversions ​ CHECKMARK CHECKMARK CHECKMARK    1. [[dataframecomplex|Complex Operations]]:​ sorts, joins, by-group operations, wide-narrow conversions ​ CHECKMARK CHECKMARK CHECKMARK
  
-   1. [[dataframeio|Input/​Output]]:​ csv files, binary IO, databases (SQL), and R data.  FIXME when csv-IO stabilizes 
  
-   --+1. Files and Operating System
  
 +   1. [[system|Operating and File System]]: command-line args, OS detection, globbing, recursive directory traversals, etc.  CHECKMARK CHECKMARK CHECKCROSS 2
  
-# More Julia ToolsPackagesApplications, and Notes+   2. [[fileio|File Input/​Output]]: ​ openingreadingwriting, and closing files. CHECKMARK ​ CHECKMARK ​ CHECKCROSS 2
  
-The next set of topics are primarily (but not exclusively) extended tools, packages, ​and applications rather than language topics.+   3. [[extpgms|External Programs]]: ​ running ​and capturing content from other programsCHECKMARK ​ CHECKCROSS ​ CHECKCROSS 8
  
  
-1. Files and Operating System+1. File Formats
  
-   1. [[system|Operating and File System]]: command-line argsOS detectionglobbingrecursive directory traversals, etc.  CHECKMARK ​CHECKMARK+   1. [[fileformats|File I/O Formats]]: e.g.CSVJSONBase64, GZip, SQL, etc. CHECKMARK ​ ​CHECKMARK ​ ​CHECKCROSS 2
  
-   2. [[fileio|File Input/​Output]]: ​ opening, reading, writing, and closing files. CHECKMARK 
  
-   3. [[extpgms|External Programs]]: ​ running and capturing content from other programs. CHECKMARK 
  
-   4. [[juliadb|JuliaDB]]:​ file-based statistical analysis on large disk-based data sets.+8Programming and Functions
  
-   --+   1. [[fundispatch|Function Dispatch ​--- Arguments and Returns]] ​ CHECKMARK CHECKMARK CHECKCROSS 5
  
 +   1. [[funother|Other Function Aspects]] --- Types of Functions. Passing by Sharing. ​ Documentation. ​ Static Variables. Macros. ​ etc.  CHECKMARK CHECKMARK CHECKCROSS 5
  
-1. Mathematical Search Algorithms+   1. [[parallel|Parallel Processing]]: ​ Basics and Embarrassingly Parallel Processing CHECKMARK ​ CHECKCROSS ​ CHECKCROSS 10
  
-   1. [[roots|Root Finding]] CHECKMARK ​CHECKMARK+   1. [[programming|Programming Tools]]: Debugging, Profiling, Benchmarking,​ Unit-Testing,​ Writing Packages.  ​CHECKMARK ​ ​CHECKCROSS ​ CHECKCROSS 9
  
-   1[[optim|Function Optimization Based on Optim.jl]] CHECKMARK CHECKMARK+   -- 
 + 
 +DataFrames and much of Julia'​s file and system I/O are not part of the core Julia language, but these capabilities are so central to Julia that this book discuss them in the core section.
  
-   1. [[jump|JuMP]] Primarily for constrained Linear and Quadratic Problems. ​ CHECKMARK CHECKMARK 
  
-   -- 
  
 +## Part 2: (More) Julia Tools, Packages, Applications,​ and Notes
  
-1Statistics ​ FIXME Tileman+WARNING Part 2 is far from done.  ​I have just started in mid-September 2018 to update to Julia 1.0.
  
-   1. [[random|Random Variables]] ​  ​CHECKMARK CHECKMARK 
  
-   2. [[unistats|Univariate Statistics]]:​ tabulatingclassifyingsummary statistics, quantiles, ​ CHECKMARK+The next set of topics are primarily (but not exclusively) extended toolspackagesand applications rather than core language topics.
  
-   3. [[unitimeseries|Univariate Time Series]]: Box-Jenkins 
  
-   4. [[multistats|Bivariate Statistics, Multivariate OLS Regressions,​ and LOESS]]: with benchmarks  ​CHECKMARK+1. [[plotting|Plotting]]: Introduction to Plots.jl CHECKMARK ​CHECKMARK
  
-   5. [[multistatsmore|More Multivariate Statistics]] ​CHECKMARK+   1. [[plotattributes ​Plot Attributes]] illustrates relevant attributes. ​ ​CHECKMARK ​CHECKMARK CHECKCROSS
  
-   6. [[classification|Clustering, Classifications]], etc.+   1. [[plotgallery2d ​2D Plot Gallery]] illustrates sample 2D plots ​CHECKMARK CHECKMARK CHECKCROSS
  
-   7. [[samplesessions|Sample Analysis Sessions]]+   1. [[plotgallery3d ​3D Plot Gallery]] illustrates sample 3D plots. ​ CHECKMARK CHECKCROSS
  
    --    --
  
  
-1[[plotting|Plotting]]:​ Introduction ​ Plots.jl ​ FIXME Tileman+2Mathematical Search Algorithms
  
-   ​1. ​Possibly not.  With [[pyplot|PyPlot]]+   1. [[roots|Root Finding]] CHECKMARK CHECKMARK CHECKMARK 1
  
-   ​2. ​Possibly not.  With [[plotlyjs|PlotlyJS]]+   2. [[optim|Function Optimization Based on Optim.jl]] CHECKMARK CHECKMARK Many
  
-   ​3. ​Possibly not.  With [[gadfly|Gadfly]] --- please avoid in 0.7.  ​will improve. +   ​3. ​FIXME: ​[[jump|JuMP]] Primarily for constrained Linear and Quadratic Problems.  ​CHECKCROSS
- +
-   4. Possibly not.  perhaps try StatPlots.jl (for integration with DataFrames, GR-based, by the same author as Plots.jl)+
  
    --    --
  
 +3. Statistics
  
-1. Finance Support ​  ​FIXME Tileman+   1. [[random|Statistical Distributions and Random Variates]] ​  ​CHECKMARK CHECKMARK CHECKMARK
  
-   1. [[findata|Prominent Free Finance Data Sources]] (EdgarFredFrenchIntrinioEtc.) +   2. [[unistats|Univariate Statistics]]: tabulatingclassifyingsummary statisticsquantiles ​CHECKMARK CHECKMARK CHECKCROSS ​2
- +
-   2. [[equities|Equities]] (Fama-Macbeth,​ Event Studies, Pfio Optimization,​ Factor-Analysis,​ Fama-French 92, etc.) +
- +
-   3. [[derivatives|Derivatives]] (Fixed Income, Options). +
- +
-   4. [[juliafin|JuliaFin]] +
- +
-   --+
  
 +   4. [[bistats|Bivariate Statistics]]:​ primarily LOESS. ​ CHECKMARK CHECKMARK CHECKMARK
  
 +   5. [[multistats|Multivariate Least-Squares Regressions]]:​ QR, SVD, GLM, and AS75, with benchmarks. ​ CHECKMARK CHECKMARK CHECKMARK
  
-1Other Language Topics+   7[[multistatsmore|Advanced Multivariate Statistics]]:​ CHECKMARK CHECKMARK CHECKMARK
  
-   1. [[baffling|Baffling Errors]]+   3FIXME [[unitimeseries|Univariate Time Series]]: Box-Jenkins,​ ARCH, GARCH, ARFIMA CHECKCROSS Package Timeseries is not yet ported to 1.0.0
  
-   1Translation Notes from RMatlabpython, perlFIXME+   8FIXME: [[classification|ClusteringClassifications]]etcCHECKCROSS
  
    --    --
  
  
 +4. Finance Support CHECKCROSS
  
-# Background+   1. FIXME: [[findata|Prominent Free Finance Data Sources]] (Edgar, Fred, French, Intrinio, Etc.) CHECKCROSS
  
-## Other Recommended Julia Docs+   2. FIXME: [[equities|Equities]] (Fama-Macbeth,​ Event Studies, Pfio Optimization,​ Factor-Analysis,​ Fama-French 92, etc.) CHECKCROSS
  
-[Official Julia Docs](https://​docs.julialang.org/​)+   3. FIXME: ​[[derivatives|Derivatives]] (Fixed Income, Options). CHECKCROSS
  
-* [Introducing_Julia](https://​en.wikibooks.org/​wiki/​Introducing_Julia)A Great Julia Wiki+   4FIXME[[juliafin|JuliaFin]] CHECKCROSS
  
-* [Learning Julia in X Minutes](https://​learnxinyminutes.com/​docs/​julia/​)+   --
  
-* [Udemy Course](https://​www.udemy.com/​julia-performing-statistical-computations/​):​ not tried. 
  
-* [Learning Julia](https://​julialang.org/​learning/​)+6Other Julia Topics CHECKCROSS
  
-[Lectures in Quantitative Economics](https://​lectures.quantecon.org/​jl/​index.html),​ Sargent Stachurski+   ​1. ​[[baffling|Baffling Errors]] CHECKMARK CHECKCROSS CHECKCROSS
  
-* [Structural Econometrics in Julia](https://​juliaeconomics.com/)+   1FIXME Translation Notes from R, Matlab, python, perl. CHECKCROSS
  
 +   --
  
-## Help, Authors, and License 
  
-* If you want to help, please email [me](mailto:​ivo.welch@gmail.com) where I got things wrong. 
  
-* This cookbook was written by [Ivo Welch](http://​www.ivo-welch.info/​),​ with some early help from [Pranav Bhat](mailto:​pranavtbhat@gmail.com). ​ [Tileman Conring](mailto:​conring@statistik.tu-dortmund.de) is coming on board, too.+---
  
-* Many of the solutions are based on help from folks at JuliaComputing and the [julia discourse forum](https://​discourse.julialang.org/​). ​ Special thanks to Tamas_Papp, Nosferican, Mohamed82008,​ Steveng, StefanK, Elrod, and others.+## META TODO
  
-* This cookbook ​is taking intellectual credit only for the organization of the examples and exposition. ​ We are merely curators. ​ Julia and its packages were written by othersand it is these authors that deserve the intellectual credit. ​ And ultimately, it is they who designed and thus they were really the only ones who could explain originally how to use their features. ​  We hope to make their efforts more worthwhile by highlighting their work and showing how to use it.+FIXME For the cookbook ​itselfwrite meta code:
  
-This wiki uses DokuWiki, with the [github-markdown](https://​guides.github.com/​features/​mastering-markdown/) syntax (and not the regular Dokuwiki syntax).+Files can be translated into multimarkdown or git-formatted-markdown ​with the following changes:
  
-* This book is free as in beer, but not free as in speech.  ​It is copyrighted by its authors.+  - we need style files for looks 
 +  - tables need to have their '​^'​ header parsed and mapped into a second line '​----:'​ type divider. 
 +  - checkmarks, checkcrosses,​ etcneed to be check, times &lt;i class="​fas fa-times"&​gt;&​lt;/​i&​gt;​ 
 +  - create a TOC 
 +  - add julia block formatters and view / paste switchers
  
 +* write a julia style highlighter/​formatter:​
  
-## TODO+  - maybe use [http://​pygments.org/​] 
 +  - flag long lines 
 +  - spaces in parenthenses should try to suggest nesting `f(..g(.h(x).)..)`. ​ Similiarly `x=.` . 
 +  - try to target comment locations towards col72, ending around col100, always use ##.
  
-FIXME For the cookbook itself, write meta code:+- improve color in links here.  not clear enough.
  
-* style formatter 
- 
-  - flag long lines 
-  - spaces in parenthenses should try to suggest nesting `f(..g(.h(x).)..)`. ​ Similiarly `x=.`. 
-  - try to put all comments at column 72 with two ##, anding around column 100 
  
 * LaTeX mapper * LaTeX mapper
Line 304: Line 216:
  
 * Copy code button * Copy code button
 +
 +* Make all const actually const
  
start.txt · Last modified: 2018/12/28 10:27 (external edit)