0

I have a pretty long JSON list that I'm looking to convert into a a data frame. I'm hoping someone can help me figure it out.

    {"body":{"overall_standings":{"years":[{"standings":null,"id":"2006"},{"standings":null,"id":"2007"},{"standings":null,"id":"2008"},{"standings":null,"id":"2009"},{"standings":null,"id":"2010"},{"standings":null,"id":"2011"},{"standings":{"teams":[{"Pitching":{"roto_points":"47.0","categories":[{"abbr":"S","roto_points":"91","value":"New York Yankees","diff":"9","rank":5},{"roto_points":"90","value":"New York Yankees","abbr":"W","diff":"7","rank":7},{"roto_points":"1383","value":"New York Yankees","abbr":"K","diff":"10","rank":4},{"abbr":"WHIP","roto_points":"1.2451","value":"New York Yankees","diff":"10","rank":4},{"abbr":"ERA","roto_points":"3.685","value":"New York Yankees","diff":"11","rank":3}]},"Total":{"behind":"0.0","roto_points":"98.0","diff":"-4.0","rank":1},"order":1,"name":"New York Yankees","Batting":{"roto_points":"51.0","categories":[{"abbr":"OBP","roto_points":"0.3371","value":"New York Yankees","diff":"7","rank":7},{"roto_points":"905","value":"New York Yankees","abbr":"RBI","diff":"10","rank":4},{"roto_points":"955","value":"New York Yankees","abbr":"R","diff":"12","rank":2},{"abbr":"SB","roto_points":"183","value":"New York Yankees","diff":"13","rank":1},{"abbr":"HR","roto_points":"247","value":"New York Yankees","diff":"9","rank":5}]},"id":"2"},{"Pitching":{"roto_points":"44.5","categories":[{"abbr":"S","roto_points":"105","value":"Los Angeles Dodgers","diff":"12","rank":2},{"roto_points":"96","value":"Los Angeles Dodgers","abbr":"W","diff":"10.5","rank":3},{"roto_points":"1410","value":"Los Angeles Dodgers","abbr":"K","diff":"11","rank":3},{"abbr":"WHIP","roto_points":"1.2798","value":"Los Angeles Dodgers","diff":"3","rank":11},{"abbr":"ERA","roto_points":"3.810","value":"Los Angeles Dodgers","diff":"8","rank":6}]},"Total":{"behind":"4.0","roto_points":"94.0","diff":"0.0","rank":2},"order":2,"name":"Los Angeles Dodgers","Batting":{"roto_points":"49.5","categories":[{"abbr":"OBP","roto_points":"0.3446","value":"Los Angeles Dodgers","diff":"11","rank":3},{"roto_points":"907","value":"Los Angeles Dodgers","abbr":"RBI","diff":"11","rank":3},{"roto_points":"909","value":"Los Angeles Dodgers","abbr":"R","diff":"9","rank":5},{"abbr":"SB","roto_points":"152","value":"Los Angeles Dodgers","diff":"11","rank":3},{"abbr":"HR","roto_points":"234","value":"Los Angeles Dodgers","diff":"7.5","rank":6}]},"id":"1"}]},"id":"2012"}]}}}

After the comma the list continues through until id 2017.

    {"standings":{"teams":[{"Pitching":{"roto_points":"40.5","categories":[{"abbr":"S","roto_points":"100","value":"Los Angeles 

Thanks in advance!

5
  • Jazzmatazz, it helps to have a json string that can be parsed, this is incomplete. If you remove the last comma from the first string and add ]}}} it makes it complete-enough for the json libraries to parse without complaining.
    – r2evans
    Oct 27, 2018 at 14:33
  • It would really help to know what you've tried so far and what you expect in the output. In this case, I suggest you manually go through (perhaps with Excel or Calc) and generate the derived column names and contents for at least a few rows. (From the looks of it, the top three levels can be dropped, and two of the lists appear to be identically-structured frames that can be reduced.)
    – r2evans
    Oct 27, 2018 at 14:38
  • I'm not familiar with parsing JSON. I didn't include the end of the data frame since it's really long. I want the columns to be id (which is the year), the other id (team id number), roto points, value, diff, rank, behind, etc.
    – Jazzmatazz
    Oct 27, 2018 at 15:12
  • You did not include a data.frame, that's a string, but the point is that it is not a complete JSON string. That's okay, I hope it's sufficient if terminated (as above). (Really long data is not typically necessary on SO, so you're fine.) Look at jsonlite and rjson.
    – r2evans
    Oct 27, 2018 at 15:15
  • I figured it would take jsonlite, I just am not sure how to use them to get the desired result.
    – Jazzmatazz
    Oct 27, 2018 at 20:09

1 Answer 1

0

This is kinda ugly. I really want somebody to come up with a better answer, especially since it keeps the nested frames, making subsetting a bit painful.

js <- jsonlite::fromJSON('{"body":{"overall_standings":{"years":[{"standings":null,"id":"2006"},{"standings":null,"id":"2007"},{"standings":null,"id":"2008"},{"standings":null,"id":"2009"},{"standings":null,"id":"2010"},{"standings":null,"id":"2011"},{"standings":{"teams":[{"Pitching":{"roto_points":"47.0","categories":[{"abbr":"S","roto_points":"91","value":"New York Yankees","diff":"9","rank":5},{"roto_points":"90","value":"New York Yankees","abbr":"W","diff":"7","rank":7},{"roto_points":"1383","value":"New York Yankees","abbr":"K","diff":"10","rank":4},{"abbr":"WHIP","roto_points":"1.2451","value":"New York Yankees","diff":"10","rank":4},{"abbr":"ERA","roto_points":"3.685","value":"New York Yankees","diff":"11","rank":3}]},"Total":{"behind":"0.0","roto_points":"98.0","diff":"-4.0","rank":1},"order":1,"name":"New York Yankees","Batting":{"roto_points":"51.0","categories":[{"abbr":"OBP","roto_points":"0.3371","value":"New York Yankees","diff":"7","rank":7},{"roto_points":"905","value":"New York Yankees","abbr":"RBI","diff":"10","rank":4},{"roto_points":"955","value":"New York Yankees","abbr":"R","diff":"12","rank":2},{"abbr":"SB","roto_points":"183","value":"New York Yankees","diff":"13","rank":1},{"abbr":"HR","roto_points":"247","value":"New York Yankees","diff":"9","rank":5}]},"id":"2"},{"Pitching":{"roto_points":"44.5","categories":[{"abbr":"S","roto_points":"105","value":"Los Angeles Dodgers","diff":"12","rank":2},{"roto_points":"96","value":"Los Angeles Dodgers","abbr":"W","diff":"10.5","rank":3},{"roto_points":"1410","value":"Los Angeles Dodgers","abbr":"K","diff":"11","rank":3},{"abbr":"WHIP","roto_points":"1.2798","value":"Los Angeles Dodgers","diff":"3","rank":11},{"abbr":"ERA","roto_points":"3.810","value":"Los Angeles Dodgers","diff":"8","rank":6}]},"Total":{"behind":"4.0","roto_points":"94.0","diff":"0.0","rank":2},"order":2,"name":"Los Angeles Dodgers","Batting":{"roto_points":"49.5","categories":[{"abbr":"OBP","roto_points":"0.3446","value":"Los Angeles Dodgers","diff":"11","rank":3},{"roto_points":"907","value":"Los Angeles Dodgers","abbr":"RBI","diff":"11","rank":3},{"roto_points":"909","value":"Los Angeles Dodgers","abbr":"R","diff":"9","rank":5},{"abbr":"SB","roto_points":"152","value":"Los Angeles Dodgers","diff":"11","rank":3},{"abbr":"HR","roto_points":"234","value":"Los Angeles Dodgers","diff":"7.5","rank":6}]},"id":"1"}]},"id":"2012"}]}}}')

It starts as a list within a list within a list, so let's get rid of the unneeded layers. (I'm showing intermediate code that is just demonstrative, no output, since it's rather verbose ... thinking I'm walking you through.)

str(js[[1]])
str(js[[1]][[1]])
str(js[[1]][[1]][[1]])
dat <- js[[1]][[1]][[1]]

On that last str, you may notice that dat$standings$teams has quite a few NULLs, let's filter them out:

dat <- Filter(length, dat$standings$teams)[[1]]

Ok, a quick helper-function and the output:

library(dplyr)
addnames <- function(x, nm, sep = "_") setNames(x, paste0(nm, sep, colnames(x)))
dat2 <- tbl_df(bind_cols(
  dat[c("id","name","order")],
  addnames(dat$Total, "Total"),
  addnames(dat$Pitching["roto_points"], "Pitching"),
  addnames(dat$Batting["roto_points"], "Batting")
)) %>%
  mutate(
    Pitching_categories = dat$Pitching$categories,
    Batting_categories = dat$Batting$categories
  ) 
dat2
# # A tibble: 2 x 11
#   id    name                order Total_behind Total_roto_points Total_diff Total_rank Pitching_roto_points Batting_roto_points Pitching_categories  Batting_categories  
#   <chr> <chr>               <int> <chr>        <chr>             <chr>           <int> <chr>                <chr>               <list>               <list>              
# 1 2     New York Yankees        1 0.0          98.0              -4.0                1 47.0                 51.0                <data.frame [5 x 5]> <data.frame [5 x 5]>
# 2 1     Los Angeles Dodgers     2 4.0          94.0              0.0                 2 44.5                 49.5                <data.frame [5 x 5]> <data.frame [5 x 5]>

This has two nested frames, so you'll to be a little creative when working with them. I suggest that this is not the best way to look at it, but it's a start. It is certainly possible (with some more elbow grease) to expand these two rows (one per team) to be ten rows (five per team, reflecting the five rows nested in each of Pitching and Batting), but that may not be what you need.

unnest(dat2)
# # A tibble: 10 x 19
#    id    name       order Total_behind Total_roto_points Total_diff Total_rank Pitching_roto_poi~ Batting_roto_po~ abbr  roto_points value     diff   rank abbr1 roto_points1 value1    diff1 rank1
#    <chr> <chr>      <int> <chr>        <chr>             <chr>           <int> <chr>              <chr>            <chr> <chr>       <chr>     <chr> <int> <chr> <chr>        <chr>     <chr> <int>
#  1 2     New York ~     1 0.0          98.0              -4.0                1 47.0               51.0             S     91          New York~ 9         5 OBP   0.3371       New York~ 7         7
#  2 2     New York ~     1 0.0          98.0              -4.0                1 47.0               51.0             W     90          New York~ 7         7 RBI   905          New York~ 10        4
#  3 2     New York ~     1 0.0          98.0              -4.0                1 47.0               51.0             K     1383        New York~ 10        4 R     955          New York~ 12        2
#  4 2     New York ~     1 0.0          98.0              -4.0                1 47.0               51.0             WHIP  1.2451      New York~ 10        4 SB    183          New York~ 13        1
#  5 2     New York ~     1 0.0          98.0              -4.0                1 47.0               51.0             ERA   3.685       New York~ 11        3 HR    247          New York~ 9         5
#  6 1     Los Angel~     2 4.0          94.0              0.0                 2 44.5               49.5             S     105         Los Ange~ 12        2 OBP   0.3446       Los Ange~ 11        3
#  7 1     Los Angel~     2 4.0          94.0              0.0                 2 44.5               49.5             W     96          Los Ange~ 10.5      3 RBI   907          Los Ange~ 11        3
#  8 1     Los Angel~     2 4.0          94.0              0.0                 2 44.5               49.5             K     1410        Los Ange~ 11        3 R     909          Los Ange~ 9         5
#  9 1     Los Angel~     2 4.0          94.0              0.0                 2 44.5               49.5             WHIP  1.2798      Los Ange~ 3        11 SB    152          Los Ange~ 11        3
# 10 1     Los Angel~     2 4.0          94.0              0.0                 2 44.5               49.5             ERA   3.810       Los Ange~ 8         6 HR    234          Los Ange~ 7.5       6

Like I said, this does not seem pretty or even elegant, but perhaps it's a good start for you, or maybe somebody else can spring-board off this for a better solution.

2
  • Thanks! It gets me exactly what I needed into a data frame. I'm certainly open to a more elegant solution, but just happy to get a solution!
    – Jazzmatazz
    Oct 28, 2018 at 16:53
  • How do I include the year as a column as well? There are additional years after 2012.
    – Jazzmatazz
    Oct 29, 2018 at 14:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.