User Tools

Site Tools


juliadb
snippet.juliarepl
julia> pkgchk( ["DataFrames" => v"0.14.1", "Loess" => v"0.3.0", "GLM" => v"0.10.1" )

Main Topic

Interestingly, JuliaDB can loadtable() compressed csv files with .csv.gz extension, too.

Please save the following to a diskfile

snippet.text
[download only julia statements]
"permno","yyyymmdd","prc","vol","ret","shrout","openprc","numtrd","retx","vwretd","ewretd","eom"
10000,19860108,-2.5,12800,-0.02439,3680,NA,NA,-0.02439,-0.020744,-0.005117,0
10000,19860109,-2.5,1400,0,3680,NA,NA,0,-0.011219,-0.011588,0
10000,19860110,-2.5,8500,0,3680,NA,NA,0,0.000083,0.003651,0
10000,19860113,-2.625,5450,0.05,3680,NA,NA,0.05,0.002749,0.002433,0
10000,19860114,-2.75,2075,0.047619,3680,NA,NA,0.047619,0.000366,0.004474,0
10000,19860115,-2.875,22490,0.045455,3680,NA,NA,0.045455,0.008206,0.007693,0
10000,19860116,-3,10900,0.043478,3680,NA,NA,0.043478,0.004702,0.00567,0
10000,19860117,-3,8470,0,3680,NA,NA,0,-0.001741,0.003297,0
10000,19860120,-3,1000,0,3680,NA,NA,0,-0.003735,-0.001355,0

Then

snippet.juliarepl
julia> using JuliaDB

julia> @time sample= loadtable("./sample.csv")
 11.862888 seconds (42.01 M allocations: 2.037 GiB, 7.64% gc time)
Table with 9 rows, 12 columns:
permno  yyyymmdd  prc     vol    ret       shrout  openprc  numtrd   retx      vwretd     ewretd     eom
────────────────────────────────────────────────────────────────────────────────────────────────────────
10000   198601082.5    128000.02439  3680    missing  missing0.024390.0207440.005117  0
10000   198601092.5    1400   0.0       3680    missing  missing  0.00.0112190.011588  0
10000   198601102.5    8500   0.0       3680    missing  missing  0.0       8.3e–5     0.003651   0
10000   198601132.625  5450   0.05      3680    missing  missing  0.05      0.002749   0.002433   0
10000   198601142.75   2075   0.047619  3680    missing  missing  0.047619  0.000366   0.004474   0
10000   198601152.875  22490  0.045455  3680    missing  missing  0.045455  0.008206   0.007693   0
10000   198601163.0    10900  0.043478  3680    missing  missing  0.043478  0.004702   0.00567    0
10000   198601173.0    8470   0.0       3680    missing  missing  0.00.001741  0.003297   0
10000   198601203.0    1000   0.0       3680    missing  missing  0.00.0037350.001355  0

julia> @time save( sample, "cutesample.jdb" )
  1.045374 seconds (4.08 M allocations: 198.137 MiB, 5.16% gc time)
Table with 9 rows, 12 columns:
permno  yyyymmdd  prc     vol    ret       shrout  openprc  numtrd   retx      vwretd     ewretd     eom
────────────────────────────────────────────────────────────────────────────────────────────────────────
10000   198601082.5    128000.02439  3680    missing  missing0.024390.0207440.005117  0
10000   198601092.5    1400   0.0       3680    missing  missing  0.00.0112190.011588  0
10000   198601102.5    8500   0.0       3680    missing  missing  0.0       8.3e–5     0.003651   0
10000   198601132.625  5450   0.05      3680    missing  missing  0.05      0.002749   0.002433   0
10000   198601142.75   2075   0.047619  3680    missing  missing  0.047619  0.000366   0.004474   0
10000   198601152.875  22490  0.045455  3680    missing  missing  0.045455  0.008206   0.007693   0
10000   198601163.0    10900  0.043478  3680    missing  missing  0.043478  0.004702   0.00567    0
10000   198601173.0    8470   0.0       3680    missing  missing  0.00.001741  0.003297   0
10000   198601203.0    1000   0.0       3680    missing  missing  0.00.0037350.001355  0

julia> @time lsample= load("./cutesample.jdb")
  0.000366 seconds (409 allocations: 20.219 KiB)
Table with 9 rows, 12 columns:
permno  yyyymmdd  prc     vol    ret       shrout  openprc  numtrd   retx      vwretd     ewretd     eom
────────────────────────────────────────────────────────────────────────────────────────────────────────
10000   198601082.5    128000.02439  3680    missing  missing0.024390.0207440.005117  0
10000   198601092.5    1400   0.0       3680    missing  missing  0.00.0112190.011588  0
10000   198601102.5    8500   0.0       3680    missing  missing  0.0       8.3e–5     0.003651   0
10000   198601132.625  5450   0.05      3680    missing  missing  0.05      0.002749   0.002433   0
10000   198601142.75   2075   0.047619  3680    missing  missing  0.047619  0.000366   0.004474   0
10000   198601152.875  22490  0.045455  3680    missing  missing  0.045455  0.008206   0.007693   0
10000   198601163.0    10900  0.043478  3680    missing  missing  0.043478  0.004702   0.00567    0
10000   198601173.0    8470   0.0       3680    missing  missing  0.00.001741  0.003297   0
10000   198601203.0    1000   0.0       3680    missing  missing  0.00.0037350.001355  0

Be aware that the load time is just for reading a few observations. JuliaDB is more of a database than an in-memory store.

Backmatter

Useful Packages on Julia Repository

Notes

*

References

juliadb.txt · Last modified: 2018/12/04 09:23 (external edit)