26

I was wondering the other day if StackOverflow had an API I could access from Mathematica, and apparently it does: "Saving plot annotations"

What's the best way to get data from StackOverflow into Mathematica? Sjoerd used the information to make a plot. I'm interested in adding SO-related notifications into a docked cell I keep in my notebooks, so I can tell when there are updates or responses without leaving Mathematica.

1
  • 1
    I learned a lot from this question and answers. Never used Mma to interface the Web before. Thanks to all Apr 21, 2011 at 16:41

2 Answers 2

27

By popular demand, the code to generate the top-10 SO answerers plot (except annotations) using the SO API (it's a pretty neat and complete API; lots of goodies there. Easy too - see my code).

Update: added App-key to ensure the code co-operates better with the SO-API (higher daily call cap). Please use it only for this app.

April 2011 enter image description here

August 2011 enter image description here

MMA 8 version! MMA7 version further down

getRepChanges[userID_Integer] :=
 Module[{totalChanges},
  totalChanges = 
   "total" /. 
    Import["http://api.stackoverflow.com/1.1/users/" <> 
      ToString[userID] <> "/reputation?key=NgVJ4Y6vFkuF-oqI-eOvOw&fromdate=0&pagesize=1&page=1",
      "JSON"
    ];
    Join @@ 
    Table[
      "rep_changes" /. 
         Import["http://api.stackoverflow.com/1.1/users/" <> 
                ToString[userID] <> 
                "/reputation?key=NgVJ4Y6vFkuF-oqI-eOvOw&fromdate=0&pagesize=100&page=" 
                <> ToString[page], 
                "JSON"
         ],
         {page, 1, Ceiling[totalChanges/100]}
    ]
  ]

topAnswerers = 
  ({"display_name","user_id", "email_hash"} /. #) & /@ 
     ("user" /. 
      ("top_users" /. 
        Import[
          "http://api.stackoverflow.com/1.1/tags/mathematica/top-answerers/all-time",    
          "JSON"
        ]
       )
      )

topAnswerers = {#, #2, 
    Import["http://www.gravatar.com/avatar/" <> #3 <> ".jpg?s=36&d=identicon&d=identicon"]
    } & @@@ topAnswerers

repChangesTopUsers =
  Table[
    repChange = 
     ReleaseHold[
        (
         Hold[
           {
              DateList["on_date" + AbsoluteTime["January 1, 1970"]], 
             "positive_rep" - "negative_rep"
           }
         ] /. #
        ) & /@ getRepChanges[userID]
      ] // Sort;
      accRepChange = {repChange[[All, 1]],Accumulate[repChange[[All, 2]]]}\[Transpose],
      {userID, topAnswerers[[All, 2]]}
    ];

pl = DateListLogPlot[
  Tooltip @@@ 
   Take[({repChangesTopUsers, Row /@ topAnswerers[[All, {3, 1}]]}\[Transpose]), 
    10], Joined -> True, Mesh -> None, ImageSize -> 1000, 
  PlotRange -> {All, {10, All}}, 
  BaseStyle -> {FontFamily -> "Arial-Bold", FontSize -> 16}, 
  DateTicksFormat -> {"MonthNameShort", " ", "Year"}, 
  GridLines -> {True, None}, 
  FrameLabel -> (Style[#, FontSize -> 18] & /@ {"Date", "Reputation", 
      "Top-10 answerers", ""})]

EDIT
Note that you can plot up to and including a top-20 by changing the value in the Take function. It gets busy pretty soon.

Tried to improve the readability of Markup code somewhat. I'm afraid this will yield some spurious spaces when copied.

EDIT
Page size back to 100 elements/page ==> fewer API calls Please note that the first call to the API is to determine the amount of posts the user has. This data is present no matter the page size, so this is preferably chosen small (10 or so, possibly 1, didn't check). Then the data is fetched in successive pages until the last page is reached. You can use the maximum page size (100) for that. Just take care that the maximum number of pages in the loop count is adjusted accordingly.

EDIT: better MMA 7 code (Fri Apr 22)

MMA 7 doesn't do JSON imports, so I do a text import instead followed by a bare-bones JSON translation. I've tested this version several times now (in MMA 8) and it seems to work without the errors I got yesterday.

getRepChanges[userID_Integer] :=
 Module[{totalChanges},
  totalChanges = 
   "total" /. 
    ImportString[
     StringReplace[(Import[
        "http://api.stackoverflow.com/1.1/users/" <> 
         ToString[userID] <> 
         "/reputation?key=NgVJ4Y6vFkuF-oqI-eOvOw&fromdate=0&pagesize=1&page=1", "Text"]), {":" ->
         "->", "[" -> "{", "]" -> "}"}], "NB"];
  Join @@ 
   Table["rep_changes" /. 
     ImportString[
      StringReplace[
       Import["http://api.stackoverflow.com/1.1/users/" <> 
         ToString[userID] <> 
         "/reputation?key=NgVJ4Y6vFkuF-oqI-eOvOw&fromdate=0&pagesize=100&page=" <> ToString[page],
         "Text"], {":" -> "->", "[" -> "{", "]" -> "}"}], 
      "NB"], {page, 1, Ceiling[totalChanges/100]}]]
topAnswerers = ({"display_name", "user_id", 
      "email_hash"} /. #) & /@ ("user" /. ("top_users" /. 
      ImportString[
       StringReplace[
        " " <> Import[
          "http://api.stackoverflow.com/1.1/tags/mathematica/top-answerers/all-time", "Text"], {":" -> "->", "[" -> "{", "]" -> "}"}], 
       "NB"]))
topAnswerers = {#, #2, 
    Import["http://www.gravatar.com/avatar/" <> #3 <> 
      ".jpg?s=36&d=identicon&d=identicon"]} & @@@ topAnswerers
repChangesTopUsers = 
  Table[repChange = 
    ReleaseHold[(Hold[{DateList[
             "on_date" + AbsoluteTime["January 1, 1970"]], 
            "positive_rep" - "negative_rep"}] /. #) & /@ 
       getRepChanges[userID]] // Sort;
   accRepChange = {repChange[[All, 1]], 
      Accumulate[repChange[[All, 2]]]}\[Transpose], {userID, 
    topAnswerers[[All, 2]]}];

DateListLogPlot[
 Tooltip @@@ 
  Take[({repChangesTopUsers, 
      Row /@ topAnswerers[[All, {3, 1}]]}\[Transpose]), 10], 
 Joined -> True, Mesh -> None, ImageSize -> 1000, 
 PlotRange -> {All, {10, All}}, 
 BaseStyle -> {FontFamily -> "Arial-Bold", FontSize -> 16}, 
 DateTicksFormat -> {"MonthNameShort", " ", "Year"}, 
 GridLines -> {True, None}, 
 FrameLabel -> (Style[#, FontSize -> 18] & /@ {"Date", "Reputation", 
     "Top-10 answerers", ""})] 

EDIT: auxiliary functions to filter on post tags These functions can be used to filter reputation gains, in order to find gains for certain tags only. tagLookup gets a post_ID integer as input and yields the specific post's tags. getQuestionIDs and getAnswerIDsFrom... go the other way. Given a tag they find all the question and answer IDs so that one can test with MemberQ whether a given post_ID belongs to this tag. Both tagLookup and getAnswerIDs are slow since many API calls are necessary. I couldn't test the last two function as either API access is down or my IP has been capped.

tagLookup[postID_Integer] :=
 Module[{im},
  im = Import["http://api.stackoverflow.com/1.1/questions/" <> ToString[postID],"JSON"];
  If[("questions" /. im) != {},
   First[("tags" /. ("questions" /. im))],
   im = Import["http://api.stackoverflow.com/1.1/answers/" <> ToString[postID],"JSON"];
   First[("tags" /. ("questions" /. Import["http://api.stackoverflow.com/1.1/questions/" <> 
          ToString[First["question_id" /. ("answers" /. im)]], "JSON"]))]
   ]
  ]

getQuestionIDs[tagName_String] := Module[{total},
  total = 
   "total" /. 
    Import["http://api.stackoverflow.com/1.1/questions?tagged=" <> 
      tagName <> "&pagesize=1", "JSON"];
  Join @@ 
   Table[("question_id" /. ("questions" /. 
        Import["http://api.stackoverflow.com/1.1/questions?key=NgVJ4Y6vFkuF-oqI-eOvOw&tagged=" <>
           tagName <> "&pagesize=100&page=" <> ToString[i], 
         "JSON"])), {i, 1, Ceiling[total/100]}]
  ]

getAnswerIDsFromQuestionID[questionID_Integer] :=
 Module[{total},
  total = 
   Import["http://api.stackoverflow.com/1.1/questions/" <> 
     ToString[questionID] <> "/answers?key=NgVJ4Y6vFkuF-oqI-eOvOw&pagesize=1", "JSON"];
  If[total === $Failed, Return[$Failed], total = "total" /. total]; 
  Join @@ Table[
    "answer_id" /. ("answers" /. 
       Import["http://api.stackoverflow.com/1.1/questions/" <> 
         ToString[questionID] <> "/answers?key=NgVJ4Y6vFkuF-oqI-eOvOw&pagesize=100&page=" <> 
         ToString[i], "JSON"]), {i, 1, Ceiling[total/100]}]
  ]

getAnswerIDsFromTag[tagName_String] :=
 Module[{},
  Join @@ (getAnswerIDsFromQuestionID /@ 
     Cases[getQuestionIDs[tagName], Except[$Failed]])
  ]
23
  • 2
    @Mr.Wizard JSON import is new in Mathematica V8. This might be useful for V7 users stackoverflow.com/questions/2633003/parsing-and-generating-json (includes code and a link to at least one other implemtation.) Apr 21, 2011 at 19:06
  • 1
    Thanks. Don't try to figure out what time zone I am on, it will only confound you.
    – Mr.Wizard
    Apr 22, 2011 at 20:47
  • 1
    @Sjoerd Applying your lessons: meta.stackexchange.com/questions/88673/… :) Apr 25, 2011 at 2:48
  • 1
    @Mr.Wizard I added a few auxiliary functions. It seemed trivial, but I encountered several difficulties. I have not been able to test everything, cause I seem to be API capped again. The additions are in JSON format as before. The above mentioned bare-bones JSON import for mma 7 should work for you. Apr 26, 2011 at 13:45
  • 1
    @Mr.Wizard The API is again up and running now. The functions find 726 questions tagged mathematica and 1707 answers. So there's a pool of 2433 mathematica-related posts that can be used to filter reputation. It's not something you want to do often as it's all pretty slow. To get the answer ids 726 individual calls to the API are necessary. This probably triggers an IP cap when you do it too often. It took 7 min on my PC. Apr 26, 2011 at 15:43
12

Brett, unrelated to SO API, but you could use RSS feed for the newest Mathematica-tagged questions. Here is my naive implementation:

QuestionHyperlink[data_] := 
 Function[{name, title, link}, 
   Hyperlink[Tooltip[title, name], link]] @@ Join[
   Cases[data, 
    XMLElement[
      "author", _, {___, XMLElement["name", {}, {name_}], ___}] :> 
     name],
   Cases[data, XMLElement["title", _, {title_}] :> title],
   Cases[data, XMLElement["link", rules_, {}] :> ("href" /. rules)]]

Cases[Import[
  "http://stackoverflow.com/feeds/tag?tagnames=mathematica&sort=\
newest", "XML"], 
 XMLElement["entry", attrs_, data_] :> 
  QuestionHyperlink[data], Infinity]

enter image description here

6
  • 1
    I recommend to use the following URL for RSS with all Wolfram-related questions: stackoverflow.com/feeds/tag/… Apr 22, 2011 at 0:46
  • @Sasha This is exactly the kind of thing I waste time doing. :) +1 for rolling your own feed. Apr 22, 2011 at 17:54
  • @Alexey Given the relatively low influx of mma questions here and the horde of answerers waiting to devour anything that's thrown in their midst, I'd love to have a solution that's instantaneous instead of based on polling. Apr 23, 2011 at 12:24
  • @Sjoerd What do you mean? Is it related to my handy RSS link? Apr 23, 2011 at 13:59
  • 1
    @Alexey My problem is, I have an RSS reader on my cellphone that I use like every other night to read about 20 feeds, but in the case of SO I would need to be pinged as soon as something arrives, otherwise the juicy question has already been thoroughly answered. Apr 23, 2011 at 17:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.