4

I need to extract the latest articles from a DBLP

The description and all the fields of all elements can be found at:

http://dblp.uni-trier.de/xml/dblp.dtd

The help file is located at:

http://dblp.uni-trier.de/xml/docu/dblpxml.pdf

SO, you have a API, you do a GET request, by Year, and you get a JSON document;

I would like to get a JSON doc with the articles from today;

But I don't know how to make the GET request using the mdate attribute;

This is the structure of a article:

<article key="journals/cacm/Szalay08"
mdate="2008-11-03">
<author>Alexander S. Szalay</author>
<title>Jim Gray, astronomer.</title>
<pages>58-65</pages>
<year>2008</year>
<volume>51</volume>
<journal>Commun. ACM</journal>
<number>11</number>
<ee>http://doi.acm.org/10.1145/
1400214.1400231</ee>
<url>db/journals/cacm/
cacm51.html#Szalay08</url>
</article>

I tryed this http://dblp.uni-trier.de/rec/bibtex/journals/acta/BayerM72 and got:

<?xml version="1.0"?>
<dblp>
<article key="journals/acta/BayerM72"
mdate="2003-11-25">
<author>Rudolf Bayer</author>
<author>Edward M. McCreight</author>
<title>Organization and Maintenance
of Large Ordered Indices</title>
...
</article>
</dblp>

I need to extract all the latest articles, by using the fields mdate.

This is a article about various requests: http://dblp.uni-trier.de/xml/docu/dblpxmlreq.pdf

the php code:

<pre>
    <?php
    $url = 'http://dblp.uni-trier.de/rec/bibtex/';
    $key = 'journals/acta/BayerM72';
    $content = file_get_contents($url . $key);
    echo $content;
    ?>
</pre>
7
  • How do you define the 'latest articles'? The example you have here will retrieve just one article. Do you expect to retrieve key by key like this, or search the entire XML database?
    – user1864610
    Apr 27, 2014 at 2:57
  • True, i extract just one article. I need to search the entire db, and the key should be mdate Apr 27, 2014 at 5:51
  • 1
    I was going to have a punt at this, since it's a relatively straightforward exercise in reading an XML file. The problem, however, is wider than it appears. The API doesn't appear to provide the sort of search you want. Searching 1.3Gb of XML online is non-starter - it'll take far too long. Searching it locally will also be too slow for most applications. This implies that you, perhaps, need to import the XML file into a database and use that to index and search. I suggest you refine the question to clarify exactly what you're trying to achieve, what you can do and what you need help with.
    – user1864610
    Apr 27, 2014 at 6:29
  • It's very simple actually; You have a API, you do a GET request, by Year, and you get a JSON document; I would like to get a JSON doc with the articles from today; But I don't know how to make the GET request using the mdate attribute; Apr 28, 2014 at 5:54
  • 1
    No where on the documentation does it say you can request articles by date. I would e-mail the site admin and make a feature request.
    – WhoIsRich
    Apr 30, 2014 at 11:05

2 Answers 2

2
+25

For parse xml there are XML Parser, XMLReader and SimpleXML. XML Parser and XMLReader are used for big file, SimpleXML - for small files (<1Mb).

function startElement($parser, $tag, $attrs) {
    global $articles, $isArticle, $i, $globTag;
    $globTag = $tag;
    if ($tag == 'article') {
        $isArticle = true;
        if (isset ( $attrs ['mdate'] )) {
            // add date from attribute in article
            $articles [$i] ['mdate'] = $attrs ['mdate'];
        }
    }
}
function endElement($parser, $tag) {
    global $articles, $isArticle, $i, $globTag;
    if ($tag == 'article') {
        $isArticle = false;
        ++ $i;
    }
}
function getElement($parser, $data) {
    global $articles, $isArticle, $i, $globTag;
    if ($isArticle) {
        $articles [$i] = $articles [$i] + [ 
                $globTag => $data 
        ];
    }
}
global $articles, $isArticle, $i, $globTag;
$articles = [ ];
$i = 0;
$isArticle = false;
$url = 'http://dblp.uni-trier.de/rec/bibtex/';
$key = 'journals/acta/BayerM72';
$url .= $key;
$parser = xml_parser_create ();

xml_set_element_handler ( $parser, "startElement", "endElement" );
xml_set_character_data_handler ( $parser, 'getElement' );
xml_parser_set_option ( $parser, XML_OPTION_CASE_FOLDING, false );

$file = fopen ( $url, 'rb' );
if ($file === false) {
    die ( "File isnt!!" );
}
$clasterSize = 8192;

while ( $data = fread ( $file, $clasterSize ) ) {
    if (! xml_parse ( $parser, $data, feof ( $file ) )) {
        die ( sprintf ( "XML error: %s at line %d", xml_error_string ( xml_get_error_code ( $parser ) ), xml_get_current_line_number ( $parser ) ) );
    }
}
xml_parser_free ( $parser );
fclose ( $file );

This is example in XML Parser.

<?php
    $url = 'http://dblp.uni-trier.de/rec/bibtex/';
    $key = 'journals/acta/BayerM72';
    $content = file_get_contents($url . $key);
$xml = new SimpleXMLElement($content);

/* Search for <dblp><article> */
$result = $xml->xpath('/dblp/article');
// $result is  an array of SimpleXMLElement objects
   var_dump($result);
    ?>

There is SimpleXML example. You get an array of SimpleXMLElement objects in result. Look a manual to get SimpleXMLElement attributes SimpleXMLElement->attributes();.

4
  • i am not interested in parsing a 1.3 gb file, i need to get the data using the api Apr 30, 2014 at 10:05
  • You will get all articls in array $articles with date. You can use SimpleXML (load file in memory)
    – Knase
    Apr 30, 2014 at 10:28
  • i have done this and it takes 15 minutes to parse the file; to download the dblp file, and then extract it, and then parse it, it is not ok Apr 30, 2014 at 10:43
  • Can you told size of the file?
    – Knase
    Apr 30, 2014 at 10:52
1

If the API doesn't offer a clean way to get updates, you'll have to cache the document and extract the articles from the changes.

DBLP2RSS, a project that creates RSS feeds from DBLP does this with a shell script:

#!/bin/sh

id="$1"
name="$2"
cache="$3"

test -d "$cache" || exit 1

curlit() {
    in-dcs && curl --proxy wwwcache.dcs.gla.ac.uk:8080 "$@" || curl --proxy "" "$@"
}

prefix="http://dblp.uni-trier.de/rec/bibtex"

echo "<dblp-content name=\"$1\">"
curlit "http://www.informatik.uni-trier.de/~ley/db/$id/index.html" 2> /dev/null | tidy -n -asxml 2> /dev/null | xml sel -N html=http://www.w3.org/1999/xhtml -t  -m '//html:a' -v '@href' -n | grep "^$name" | while read path; do 
    # Should cache here
    cachefile="$cache/$id/$path"

    if ! test -f "$cachefile"; then
    mkdir -p "$(dirname $cachefile)"
    curlit "http://www.informatik.uni-trier.de/~ley/db/$id/$path" 2> /dev/null | tidy -n -asxml 2> /dev/null >  $cachefile
    echo "Got $cachefile"

    fi
    cat "$cachefile" | xml sel -N html=http://www.w3.org/1999/xhtml -t  -m '//html:a' -v '@href' -n | egrep '^'"$prefix"'.*\.xml$' | sed -e 's#^'"$prefix"/'#<dblpkey>#' -e 's/\.xml$/<\/dblpkey>/'

done 

echo "</dblp-content>"

This doesn't get the articles, but you could take the same approach.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.