Julia Programming - Working with Datasets

In this chapter, we shall discuss in detail about datasets.

CSV files

As we know that CSV (Comma Separated Value) file is a plain text file which uses commas to separate fields and values of those fields. The extension of these files is .CSV. We have various methods provided by Julia programming language to perform operations on CSV files.

Import a .CSV file in Julia

To import a .CSV file, we need to install CSV package. Use the following command to do so −

using pkg
pkg.add("CSV")

Reading data

To read data from a CSV file in Julia we need to use read() method from CSV package as follows −

julia> using CSV
julia> CSV.read("C://Users//Leekha//Desktop//Iris.csv")
150×6 DataFrame
│ Row │  Id   │ SepalLengthCm │ SepalWidthCm │ PetalLengthCm │ PetalWidthCm │ Species        │
│     │ Int64 │      Float64  │     Float64  │     Float64   │    Float64   │ String         │
├─────┼───────┼───────────────┼──────────────┼───────────────┼──────────────┼─────────-------┤
│  1  │   1   │      5.1      │     3.5      │     1.4       │    0.2       │ Iris-setosa    │
│  2  │   2   │      4.9      │     3.0      │     1.4       │    0.2       │ Iris-setosa    │
│  3  │   3   │      4.7      │     3.2      │     1.3       │    0.2       │ Iris-setosa    │
│  4  │   4   │      4.6      │     3.1      │     1.5       │    0.2       │ Iris-setosa    │
│  5  │   5   │      5.0      │     3.6      │     1.4       │    0.2       │ Iris-setosa    │
│  6  │   6   │      5.4      │     3.9      │     1.7       │    0.4       │ Iris-setosa    │
│  7  │   7   │      4.6      │     3.4      │     1.4       │    0.3       │ Iris-setosa    │
│  8  │   8   │      5.0      │     3.4      │     1.5       │    0.2       │ Iris-setosa    │
│  9  │   9   │      4.4      │     2.9      │     1.4       │    0.2       │ Iris-setosa    │
│  10 │   10  │      4.9      │     3.1      │     1.5       │    0.1       │ Iris-setosa    │
⋮
│ 140 │ 140   │      6.9      │     3.1      │     5.4       │    2.1       │ Iris-virginica │
│ 141 │ 141   │      6.7      │     3.1      │     5.6       │    2.4       │ Iris-virginica │
│ 142 │ 142   │      6.9      │     3.1      │     5.1       │    2.3       │ Iris-virginica │
│ 143 │ 143   │      5.8      │     2.7      │     5.1       │    1.9       │ Iris-virginica │
│ 144 │ 144   │      6.8      │     3.2      │     5.9       │    2.3       │ Iris-virginica │
│ 145 │ 145   │      6.7      │     3.3      │     5.7       │    2.5       │ Iris-virginica │
│ 146 │ 146   │      6.7      │     3.0      │     5.2       │    2.3       │ Iris-virginica │
│ 147 │ 147   │      6.3      │     2.5      │     5.0       │    1.9       │ Iris-virginica │
│ 148 │ 148   │      6.5      │     3.0      │     5.2       │    2.0       │ Iris-virginica │
│ 149 │ 149   │      6.2      │     3.4      │     5.4       │    2.3       │ Iris-virginica │
│ 150 │ 150   │      5.9      │     3.0      │     5.1       │    1.8       │ Iris-virginica │

Creating new CSV file

To create new CSV file, we need to use touch()command from CSV package. We also need to use DataFrames package to write the newly created content to new CSV file −

julia> using DataFrames
julia> using CSV
julia> touch("1234.csv")
"1234.csv"

julia> new = open("1234.csv", "w")
IOStream(<file 1234.csv>)

julia> new_data = DataFrame(Name = ["Gaurav", "Rahul", "Aarav", "Raman", "Ravinder"],
                  RollNo = [1, 2, 3, 4, 5],
                  Marks = [54, 67, 90, 23, 95])
                  
5×3 DataFrame
│ Row │  Name    │ RollNo │ Marks │
│     │  String  │ Int64  │ Int64 │
├─────┼──────────┼────────┼───────┤
│ 1   │  Gaurav  │   1    │   54  │
│ 2   │  Rahul   │   2    │   67  │
│ 3   │   Aarav  │   3    │   90  │
│ 4   │   Raman  │   4    │   23  │
│ 5   │ Ravinder │   5    │   95  │

julia> CSV.write("1234.csv", new_data)
"1234.csv"

julia> CSV.read("1234.csv")
5×3 DataFrame
│ Row │    Name  │ RollNo │ Marks │
│     │  String  │ Int64  │ Int64 │
├─────┼──────────┼────────┼───────┤
│   1 │   Gaurav │   1    │   54  │
│   2 │   Rahul  │   2    │   67  │
│   3 │   Aarav  │   3    │   90  │
│   4 │   Raman  │   4    │   23  │
│   5 │ Ravinder │   5    │   95  │

HDF5

The full form of HDF5 is Hierarchical Data Format v5. Following are some of its properties −

  • A “group” is similar to a directory, a “dataset” is like a file.

  • To associate metadata with a particular group, it uses attributes.

  • It uses ASCII names for different objects.

  • Language wrappers are often known as “low level” or “high level”.

Opening HDF5 files

HDF5 files can be opened with h5open command as follows −

fid = h5open(filename, mode)

Following table describes the mode −

Sl.No Mode & Meaning
1

"r"

read-only
2

"r+"

read-write − It will preserve any existing contents.
3

"cw"

read-write − It will create file if not existing.

It will also preserve existing contents.
4

"w"

read-write − It will destroy any existing contents.

The above command will produce an object of type HDF5File and a subtype of the abstract type DataFile.

Closing HDF5 files

Once finished with a file, we should close it as follows −

close(fid)

It will also close all the objects in the file.

Opening HDF5 objects

Suppose if we have a file object named fid and it has a group called object1, it can be opened as follows −

Obj1 = fid[“object1”]

Closing HDF5 objects

close(obj1)

Reading data

A group “g” containing a dataset with path “dtset” and we have opened dataset as dset1 = g[dtset]. We can read the information in following ways −

ABC = read(dset1)
ABC = read(g, "dtset")
Asub = dset1[2:3, 1:3]

Writing data

We can create the dataset as follows −

g["dset1"] = rand(3,5)
write(g, "dset1", rand(3,5))

XML files

Here we will be discussing about LightXML.jl package which is a light-weight Julia wrapper for libxml2. It provides the following functionalities −

  • Parsing an XML file

  • Accessing XML tree structure

  • Creating an XML tree

  • Exporting an XML tree to a string

Example

Suppose we have an xml file named new.xml as follows −

<Hello>
      <to>Gaurav</to>
      <from>Rahul</from>
      <heading>Reminder to meet</heading>
      <body>Friend, Don't forget to meet this weekend!</body>
</Hello>

Now, we can parse this file by using LightXML as follows −

julia> using LightXML
#below code will parse this xml file
julia> xdoc = parse_file("C://Users//Leekha//Desktop//new.xml")
<?xml version="1.0" encoding="utf-8"?>
<Hello>
<to>Gaurav</to>
<from>Rahul</from>
<heading>Reminder to meet</heading>
<body>Friend, Don't forget to meet this weekend!</body>
</Hello>

Following example explains how to get the root element −

julia> xroot = root(xdoc);
julia> println(name(xroot))
Hello
#Traversing all the child nodes and also print element names
julia> for c in child_nodes(xroot) # c is an instance of XMLNode
            println(nodetype(c))
            if is_elementnode(c)
               e = XMLElement(c) # this makes an XMLElement instance
               println(name(e))
            end
         end
3
1
to
3
1
from
3
1
heading
3
1
body
3

RDatasets

Julia has RDatasets.jl package providing easy way to use and experiment with most of the standard data sets which are available in the core of R. To load and work with one of the datasets included in RDatasets packages, we need to install RDatasets as follows −

julia> using Pkg
julia> Pkg.add("RDatasets")

Subsetting the data

For example, we will use the Gcsemv dataset in mlmRev group as follows −

julia> GetData = dataset("mlmRev","Gcsemv");
julia> summary(GetData);
julia> head(GetData)
6×5 DataFrame
│ Row │     School   │     Student  │     Gender   │  Written │   Course │
│     │ Categorical… │ Categorical… │ Categorical… │ Float64⍰ │ Float64⍰ │
├─────┼──────────────┼──────────────┼──────────────┼──────────┼──────────┤
│   1 │     20920    │      16      │      M       │  23.0    │  missing │
│   2 │     20920    │      25      │      F       │  missing │   71.2   │
│   3 │     20920    │      27      │      F       │  39.0    │   76.8   │
│   4 │     20920    │      31      │      F       │  36.0    │   87.9   │
│   5 │     20920    │      42      │      M       │  16.0    │   44.4   │
│   6 │     20920    │      62      │      F       │  36.0    │  missing │

We can select the data for a particular school as follows −

julia> GetData[GetData[:School] .== "68137", :]
104×5 DataFrame
│ Row │     School   │     Student  │     Gender   │  Written │   Course │
│     │ Categorical… │ Categorical… │ Categorical… │ Float64⍰ │ Float64⍰ │
├─────┼──────────────┼──────────────┼──────────────┼──────────┼──────────┤
│  1  │     68137    │      1       │     F        │   18.0   │   56.4   │
│  2  │     68137    │      2       │     F        │   23.0   │   55.5   │
│  3  │     68137    │      3       │     F        │   25.0   │  missing │
│  4  │     68137    │      4       │     F        │   29.0   │   73.1   │
│  5  │     68137    │      5       │     F        │  missing │   66.6   │
│  6  │     68137    │      9       │     F        │   20.0   │   60.1   │
│  7  │     68137    │     11       │     F        │   34.0   │   63.8   │
│  8  │     68137    │     12       │     F        │   60.0   │   89.8   │
│  9  │     68137    │     13       │     F        │   44.0   │   76.8   │
│  10 │     68137    │     14       │     F        │   20.0   │   58.3   │
⋮
│ 94  │     68137    │     252      │     M        │  missing │   75.9   │
│ 95  │     68137    │     254      │     M        │     35.0 │ missing  │
│ 96  │     68137    │     255      │     M        │     36.0 │   62.0   │
│ 97  │     68137    │     258      │     M        │     23.0 │   61.1   │
│ 98  │     68137    │     260      │     M        │     25.0 │ missing  │
│ 99  │     68137    │     261      │     M        │     46.0 │    89.8  │
│ 100 │     68137    │     264      │     M        │     50.0 │    70.3  │
│ 101 │     68137    │     268      │     M        │     15.0 │    43.5  │
│ 102 │     68137    │     270      │     M        │  missing │    73.1  │
│ 103 │     68137    │     272      │     M        │     43.0 │    78.7  │
│ 104 │     68137    │     273      │     M        │     35.0 │    60.1  │

Sorting the data

With the help of sort!() function, we can sort the data. For example, here we will sort the dataset in ascending examination scores −

julia> sort!(GetData, cols=[:Written])
1905×5 DataFrame
│ Row  │       School │      Student │       Gender │  Written │   Course │
│      │ Categorical… │ Categorical… │ Categorical… │ Float64⍰ │ Float64⍰ │
├──────┼──────────────┼──────────────┼──────────────┼──────────┼──────────┤
│  1   │    22710     │       77     │     F        │    0.6   │   41.6   │
│  2   │    68137     │       65     │     F        │    2.5   │   50.0   │
│  3   │    22520     │       115    │     M        │    3.1   │   9.25   │
│  4   │    68137     │       80     │     F        │    4.3   │   50.9   │
│  5   │    68137     │       79     │     F        │    7.5   │   27.7   │
│  6   │    22710     │       57     │     F        │    11.0  │   73.1   │
│  7   │    64327     │       19     │     F        │    11.0  │   87.0   │
│  8   │    68137     │       85     │     F        │    11.0  │   27.7   │
│  9   │    68137     │       97     │     F        │    11.0  │   57.4   │
│ 10   │    68137     │       100    │     F        │    11.0  │   61.1   │
⋮
│ 1895 │    74874     │       83     │     F        │ missing  │    81.4  │
│ 1896 │    74874     │       86     │     F        │ missing  │    92.5  │
│ 1897 │    76631     │       79     │     F        │ missing  │    84.2  │
│ 1898 │    76631     │       193    │     M        │ missing  │    72.2  │
│ 1899 │    76631     │       221    │     F        │ missing  │    76.8  │
│ 1900 │    77207     │       5001   │     F        │ missing  │    82.4  │
│ 1901 │    77207     │       5062   │     M        │ missing  │    75.0  │
│ 1902 │    77207     │       5063   │     F        │ missing  │    79.6  │
│ 1903 │    84772     │       17     │     M        │ missing  │    88.8  │
│ 1904 │    84772     │       49     │     M        │ missing  │    74.0  │
│ 1905 │    84772     │       85     │     F        │ missing  │    90.7  │

Statistics in Julia

To work with statistics, Julia has StatsBase.jl package providing easy way to do simple statistics. To work with statistics, we need to install StatsBase package as follows −

julia> using Pkg
julia> Pkg.add("StatsBase")

Simple Statistics

Julia provides methods to define weights and calculate mean.

We can use weights() function to define weights vectors as follows −

julia> WV = Weights([10.,11.,12.])
3-element Weights{Float64,Float64,Array{Float64,1}}:
 10.0
 11.0
 12.0

You can use the isempty() function to check whether the weight vector is empty or not −

julia> isempty(WV)
false

We can check the type of weight vectors with the help of eltype() function as follows −

julia> eltype(WV)
Float64

We can check the length of the weight vectors with the help of length() function as follows −

julia> length(WV)
3

There are different ways to calculate the mean −

  • Harmonic mean − We can use harmmean() function to calculate the harmonic mean.

julia> A = [3, 5, 6, 7, 8, 2, 9, 10]
8-element Array{Int64,1}:
 3
 5
 6
 7
 8
 2
 9
 10
julia> harmmean(A)
4.764831009217679

  • Geometric mean − We can use geomean() function to calculate the Geometric mean.

julia> geomean(A)
5.555368605381863

  • General mean − We can use mean() function to calculate the general mean.

julia> mean(A)
6.25

Descriptive Statistics

It is that discipline of statistics in which information is extracted and analyzed. This information explains the essence of data.

Calculating variance

We can use var() function to calculate the variance of a vector as follows −

julia> B = [1., 2., 3., 4., 5.];
julia> var(B)
2.5

Calculating weighted variance

We can calculate the weighted variance of a vector x w.r.t to weight vector as follows −

julia> B = [1., 2., 3., 4., 5.];
julia> a = aweights([4., 2., 1., 3., 1.])
5-element AnalyticWeights{Float64,Float64,Array{Float64,1}}:
 4.0
 2.0
 1.0
 3.0
 1.0
julia> var(B, a)
2.066115702479339

Calculating standard deviation

We can use std() function to calculate the standard variation of a vector as follows −

julia> std(B)
1.5811388300841898

Calculating weighted standard deviation

We can calculate the weighted standard deviation of a vector x w.r.t to weight vector as follows −

julia> std(B,a)
1.4373989364401725

Calculating mean and standard deviation

We can calculate the mean and standard deviation in a single command as follows −

julia> mean_and_std(B,a)
(2.5454545454545454, 1.4373989364401725)

Calculating mean and variance

We can calculate the mean and variance in a single command as follows −

julia> mean_and_var(B,a)
(2.5454545454545454, 2.066115702479339)

Samples and Estimations

It may be defined as the discipline of statistics where, for analysis, sample units will be selected from a large population set.

Following are the ways in which we can do sampling −

Taking random samples is the simplest way of doing sampling. In this we draw a random element from the array, i.e., the population set. The function for this purpose is sample().

Example

julia> A = [8.,12.,23.,54.5]
4-element Array{Float64,1}:
 8.0
 12.0
 23.0
 54.5
julia> sample(A)
12.0

Next, we can take “n” elements as random samples.

Example

julia> A = [8.,12.,23.,54.5]
4-element Array{Float64,1}:
 8.0
 12.0
 23.0
 54.5
julia> sample(A, 2)
2-element Array{Float64,1}:
 23.0
 54.5

We can also write the sampled elements to pre-allocated elements of length “n”. The function to do this task is sample!().

Example

julia> B = [1., 2., 3., 4., 5.];
julia> X = [2., 1., 3., 2., 5.];
julia> sample!(B,X)
5-element Array{Float64,1}:
 2.0
 2.0
 4.0
 1.0
 3.0

Another way is to do direct sampling which will randomly picks the numbers from a population set and stores them in another array. The function to do this task is direct_sample!().

Example

julia> StatsBase.direct_sample!(B, X)
5-element Array{Float64,1}:
 1.0
 4.0
 4.0
 4.0
 5.0

Knuth’s algorithms is one other way in which random sampling is done without replcement.

Example

julia> StatsBase.knuths_sample!(B, X)
5-element Array{Float64,1}:
 5.0
 3.0
 4.0
 2.0
 1.0
Advertisements