2

I want to use Julia DataFrames to construct a 3x3 Markov transition matrix i.e. a frequency matrix that tells me the likelihood of transitioning from each of 3 states to the others. I am trying to learn data frames and I would like to learn the best way to do this. This is more for general learning than about this particular example.

Here's some code I tried so far with some example data but I am not really familiar enough with how to think about dataframes to know how to proceed.

Any suggestions? Thank you.


state=[2,2,3,1,1,3,3,2,1,1,3,1,2,3,2,3,1,2,3,3,1]
statelag=[1,2,2,3,1,1,3,3,2,1,1,3,1,2,3,2,3,1,2,3,3]
df = DataFrame(state=state, statelag=statelag)

markov = combine(groupby(df, [:statelag, :state]), nrow => :cat_countmar)
sort!(markov, :statelag, :state) # this gives the number of occurences of each tranistion

total = combine(groupby(df, :statelag), nrow => :cat_count) 
# this gives the number of occurences of each state


trans = Array{Float64}(undef, (3,3))
# trans should give probability of transitioning between different states

I need to basically "divide" catcountmar of by cat_count so that I'm dividing the number of occurrences of a transition from state i to state j by the number of occurences of state i. This will give the desired transition frequency. But I don't see how to put markov and total together in one data frame and easily carry out this computation.

1 Answer 1

3

You can use the transform function to get results of the same length of your original dataframe. See the code below for an example.

state=[2,2,3,1,1,3,3,2,1,1,3,1,2,3,2,3,1,2,3,3,1]
statelag=[1,2,2,3,1,1,3,3,2,1,1,3,1,2,3,2,3,1,2,3,3]
df = DataFrame(state=state, statelag=statelag)

df = transform(groupby(df, [:statelag, :state]), nrow => :cat_countmar)
# get the number of occurrences for each transition

df = transform(groupby(df, :statelag), nrow => :cat_count) 
# get the number of occurrences for each state

df[:,:prob] = df[:,:cat_countmar]./df[:,:cat_count]
# get the transition probability 

trans = unique(df[:,[:statelag,:state,:prob]])
# remove unnecessary rows and columns  

reshape_trans = unstack(trans,:state,:prob)
# Transform into a matrix format

trans_mat = convert(Matrix{Float64},reshape_trans[:,2:4])
# Finally, convert the dataframe to a matrix

Final Output:

3×3 Array{Float64,2}:
 0.285714  0.428571  0.285714
 0.166667  0.166667  0.666667
 0.5       0.25      0.25
Sign up to request clarification or add additional context in comments.

2 Comments

I'm not super clear on what exactly the unstack function is doing here. If you could elaborate on that, that would be really helpful. I had a look at the docs for that function and was still confused, but if you insist on rtfm I'll give it another shot :) thanks
The unstack is equivalent to reshape in other statistical softwares (like stata). It transfers a long panel to a wider (but shorter) panel. You do not necessarily need to do the last two steps as the trans dataframe should give all the information you need.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.