5

I am trying to extract citations from a PDF. I confirmed that my regex worked on Rubular here but when I test my code on a real PDF it spits out some oddly spaced and wrong information. How can I fix this regex so it only extracts APA paper citations (the ones in the references section, not in-text). The APA Examples might be useful. I am trying to get references from a research paper. If you need more details please let me know. Multiple regex are acceptable for this answer, but I do need to be able to extract author, title, date, and journal. My attempt is below if it helps anybody:

require 'pdf-reader'
io = open('https://vhil.stanford.edu/pubs/2007/yee-proteus-effect.pdf')
out=open('dump.txt',"w")
reader = PDF::Reader.new(io)

reader.pages.each do |page|
    /([a-zA-Z.,&\s]+?)(\(\d+\)).([\sa-zA-Z,:\n\t]+).([\sa-zA-Z,]+).([\sa-zA-Z,]+)/.match(page.text){|m|
        puts "===CITATION===="
        puts "author: "+m[0].to_str.gsub(/\n\r\t/,'')
        puts "title: "+m[2].to_str.gsub(/\n\r\t/,'')
        puts "date: "+m[1].to_str.gsub(/\n\r\t/,'')
        puts "journal: "+m[3].to_str.gsub(/\n\r\t/,'')
  }
  #puts page.raw_content
end
puts"\n\n\n=======\n\n\n======"
puts reader.pages.last

More examples (in response to comments) here and here THE ENTIRE PAPERhere

To get these examples I ran out.puts page.text inside my foreach loop. Then I copied chunks of text into Rubular and tested using my original regex (above).

6
  • 1
    Never minding the regex: how do you know your input is correct? The "odd spaced and wrong" text may very well come out that way because trying to get the original text out of a PDF is a black art, and can fail for many reasons.
    – Jongware
    Sep 28, 2015 at 0:53
  • I am writing the contents of the file to a text file to check this problem in my code. The text file seems accurate, although it may not be
    – Rilcon42
    Sep 28, 2015 at 1:08
  • Can you include a link to such a text file so we are sure to start with the same input? (In addition, remove the pdf from your question as it does not sound to be relevant in this case.)
    – Jongware
    Sep 28, 2015 at 6:35
  • Would you like me to post the PDF as text, or my output when I run the code above?
    – Rilcon42
    Sep 28, 2015 at 18:24
  • Please post (not too big) into your question examples of input text that causes you problems.
    – Bohemian
    Sep 28, 2015 at 23:42

3 Answers 3

8
+50

EDIT

Try this one, I tried to modify it in such way, to match positions from your comment.

^(?<author>[A-Z](?:(?!$)[A-Za-z\s&.,'’])+)\((?<year>\d{4})\)\.?\s*(?<title>[^()]+?[?.!])\s*(?:(?:(?<jurnal>(?:(?!^[A-Z])[^.]+?)),\s*(?<issue>\d+)[^,.]*(?=,\s*\d+|.\s*Ret))|(?:In\s*(?<editors>[^()]+))\(Eds?\.\),\s*(?<book>[^().]+)|(?:[^():]+:[^().]+\.)|(?:Retrieved|Paper presented))

Rebular DEMO
Regex101 DEMO

The last part (?:Retrieved|Paper presented) could be extended with other word which could occur after title.

This regex consist two main parts:

  1. ^(?<author>[A-Z](?:(?!$)[A-Za-z\s&.,'’])+)\((?<year>\d{4})\)\.?\s*(?<title>[^()]+?[?.!])\s* the shared part which is matching authors, date and title:

    • (?<author>[A-Z](?:(?!$)[A-Za-z\s&.,'’])+) - author group, match from beginning of a line, only if word starts from capital letter, to avoid matching somewhere in the text, nod in references, followed by letters, whitespaces, some punctuation, etc. You can always add some symbols to [A-Za-z\s&.,'’] character class, if there is more character, which can occur in this part of reference.
    • \((?<year>\d{4})\) - year group capture digits, if they are inside brackets,
    • (?<title>[^()]+?[?.!])\s* - title group capture one of more of any characters, but not parentheses, followed by character from character class [?.!], I used [^()] because during tests I found that its prevents regex from multiline invalid matching, what also important, this part matching is restricted by alternatives, without them, it would give invalid results, so it is not matching independently separate parts,
  2. (?:(?:(?<jurnal>(?:(?!^[A-Z])[^.]+?)),\s*(?<issue>\d+)[^,.]*(?=,\s*\d+|.\s*Ret))|(?:In\s*(?<editors>[^()]+))\(Eds?\.\),\s*(?<book>[^().]+)|(?:[^():]+:[^().]+\.)|(?:Retrieved|Paper presented)) alternatives for matching rest of content.

    • (?<jurnal>(?:(?!^[A-Z])[^.]+?)),\s*(?<issue>\d+)[^,.]*(?=,\s*\d+|.\s*Ret)) - this alternative match jurnal title nad issue. jurnal group matches the fragment beginning with capital letter, followed by not points (not .), with lazy quantifire (match just as much as it is necessary to succeed) and followed by comma. The [^.] is used, because some jurnal titles consist commas, I couldn't use [^,] which was my first idea, so I restreined matching of this part with point, which occour always in the end of reference, with reluctant quantifire, it allaws to give up already matched fragments (up to point) for following matching. The issue group match digits with some content up to next comma or point, if it is followed by comma with digits (pages numbers) or point and word retrived,
    • (?:In\s*(?<editors>[^()]+))\(Eds?\.\),\s*(?<book>[^().]+) - this part match reference to edited book, editor group match anything but not paratheses (up to Eds. or Ed. key word) followed by title restricted by next paratheses with pages numbers or point
    • (?:[^():]+:[^().]+\.) - this part is used to match references only with information about publisher and place of publication, the previous approuch with using a ? on whole alteratives part was not effective, because it matched also in places where another alternative should be matched,
    • (?:Retrieved|Paper presented)) - this part is for matching references which refer to online source or presentation, etc. I could be extended with other keywords,

old attempts

If you need only author, date, title and jurnal with issue, you can try with:

^(?<author>(?:(?!$)[A-Za-z\s&.,'’])+)\((?<year>\d{4})\)\.?\s*(?<title>[^?.!]+?[.?!])\s*(?!\s*Retrieved)(?:(?:(?<jurnal>(?:(?!^[A-Z])[^,])+?),\s*(?<issue>\d+)))

DEMO rebular
DEMO regex101

however if you are interested also in edited books, try with:

^(?<author>(?:(?!$)[A-Za-z\s&.,'’])+)\((?<year>\d{4})\)\.?\s*(?<title>[^?.!]+?[.?!])\s*(?:(?<retrieved>[Rr]etrieved.+)|(?:(?:(?<jurnal>(?:(?!^[A-Z])[^,])+?),\s*(?<issue>\d+)))|\s*In(?<editors>[^\(]+)\(Eds\.\),(?<book>[^.()]+))?

DEMO rebular
DEMO regex101

Both regular expression will capture relevant values into named groups: author, year, title, etc.

3
  • This comes very close to being perfect. Using the first regex you suggest match 1 (using the entire paper linked above) fails. Every other match looks good. using the second regex matches 52,54, and 55 fail.
    – Rilcon42
    Oct 1, 2015 at 3:13
  • I'm sorry, but I can't understand, why re.compile("^(?<author>[A-Z](?:(?!$)[A-Za-z\s&.,'’])+)\((?<year>\d{4})\)\.?\s*(?<title>[^()]+?[?.!])\s*(?:(?:(?<jurnal>(?:(?!^[A-Z])[^.]+?)),\s*(?<issue>\d+)[^,.]*(?=,\s*\d+|.\s*Ret))|(?:In\s*(?<editors>[^()]+))\(Eds?\.\),\s*(?<book>[^().]+)|(?:[^():]+:[^().]+\.)|(?:Retrieved|Paper presented))") doesn't work? error: unknown extension ?<a at position 2 Nov 10, 2020 at 7:18
  • @ЛюбовьПономарева it seems that you try to run in in Python, am I right? that regex in in Ruby syntax, so there are some differences. Try to add P before named groups like here
    – m.cekiera
    Nov 22, 2020 at 8:06
0

I'll take a shot:

/^(.+?)\s*(\(\d.+?\))\.?\s*(.+?)\.\s*(.+?),\s\d/m
1
  • Thanks for trying, however your regex contained the paper header (which was repeated at the top of every page). Some of the regexes also contained large passages from the paper.
    – Rilcon42
    Sep 29, 2015 at 3:05
0

This seems to work for all citations in the (previously) linked document:

^([\S\s]*?)\s*\((\d{4})\)\.?\s*([\S\s]*?\??)(?:(?:[.]|(?<=\? ))(?=(?:[^.]|[.](?:[\d,]|com|pdf|html?))*?[. ]*$)\s*|(?= Retrieved ))(.*)?

See live demo.

This allows for newlines within citations, except within the journal text.

3
  • When I tested this with the entire paper it only returned 3 citations, the first citation on each page. When I saved the entire paper as a string it only returned the first citation in the string.
    – Rilcon42
    Oct 1, 2015 at 2:53
  • I have changed the regex so there can be embedded linefeeds. See edit.
    – Bohemian
    Oct 1, 2015 at 2:59
  • When I test with the entire paper (which I just added as a Rubular link in my post) this occasionally picks up large chunks of text
    – Rilcon42
    Oct 1, 2015 at 3:06

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.