Given page 4 of this PDF file, I was wondering if there is a way (using any R library) to import the 2 columns named SCALE SCORE
and FREQ.
into R as a .csv
or other R-friendly formats?
-
1@Reza There is nothing harsh in me basically quoting the official SO documentation: Please take the tour: "Focus on questions about an actual problem you have faced. Include details about what you have tried and exactly what you are trying to do. [...] show your work!". Cheers.– HenrikJul 5, 2020 at 20:20
2 Answers
I have written a package that can help extract text from pdfs. It's written from scratch in C++ and is fairly fast (usually a bit faster than pdftools). At the moment you still need to wrangle the text into a table - as you would in pdftools. In your case, it would work like this:
library(dplyr)
library(PDFR)
df <- pdfpage("C:/users/Administrator/Documents/sales.pdf", 4)
df <- df[df$left > 440,] %>%
group_by(top) %>%
arrange(left, by_group = TRUE) %>%
summarize(text = paste(text, collapse = ",")) %>%
arrange(-top) %>%
filter(seq(nrow(.)) > 4) %>%
`[[`(2) %>%
read.csv(text = ., header = FALSE,
col.names = c("freq", "cum_freq", "perc", "cum_perc"))
Which gives you:
#> freq cum_freq perc cum_perc
#> 1 142 142 0.04 0.04
#> 2 15 157 0.00 0.04
#> 3 78 235 0.02 0.06
#> 4 269 504 0.07 0.13
#> 5 840 1344 0.21 0.34
#> 6 1690 3034 0.42 0.76
#> 7 3254 6288 0.81 1.57
#> 8 5413 11701 1.35 2.92
#> 9 7659 19360 1.91 4.83
#> 10 9696 29056 2.42 7.24
#> 11 11529 40585 2.87 10.12
#> 12 13145 53730 3.28 13.39
#> 13 13830 67560 3.45 16.84
#> 14 14844 82404 3.70 20.54
#> 15 15153 97557 3.78 24.32
#> 16 15120 112677 3.77 28.09
#> 17 15347 128024 3.83 31.92
#> 18 15525 143549 3.87 35.79
#> 19 15710 159259 3.92 39.70
#> 20 15596 174855 3.89 43.59
#> 21 15529 190384 3.87 47.46
#> 22 15451 205835 3.85 51.31
#> 23 15259 221094 3.80 55.12
#> 24 15028 236122 3.75 58.86
#> 25 15147 251269 3.78 62.64
#> 26 14683 265952 3.66 66.30
#> 27 14469 280421 3.61 69.91
#> 28 14229 294650 3.55 73.45
#> 29 13523 308173 3.37 76.82
#> 30 13246 321419 3.30 80.13
#> 31 12987 334406 3.24 83.36
#> 32 12264 346670 3.06 86.42
#> 33 11964 358634 2.98 89.40
#> 34 10841 369475 2.70 92.11
#> 35 9958 379433 2.48 94.59
#> 36 8529 387962 2.13 96.72
#> 37 6729 394691 1.68 98.39
#> 38 4437 399128 1.11 99.50
#> 39 2010 401138 0.50 100.00
Although this may seem a bit involved, it is great for pdfs like yours where the tables are the same on each page. If you ran the above code inside an lapply
loop it could get multiple pages at a time far more quickly than cutting and pasting would.
To install you need devtools:
install.packages("devtools")
devtools::install_github("AllanCameron/PDFR")
Edit
If there are installation problems, here is the equivalent in pdftools:
install.packages(pdftools)
df <- pdftools::pdf_data("https://tea.texas.gov/sites/default/files/Scale%20Score%20Distribution%20Graph%201_Grade%203%20to%208%20English-r2_tagged.pdf")[[4]]
df <- df[df$x > 440,] %>%
group_by(y) %>%
arrange(x, by_group = TRUE) %>%
summarize(text = paste(text, collapse = ",")) %>%
arrange(y) %>%
`[[`(2) %>%
`[`(3:41) %>%
read.csv(text = ., header = FALSE,
col.names = c("freq", "cum_freq", "perc", "cum_perc"))
-
Thanks for sharing your package. I had been trying
tabulizer
, but bleh,rJava
. Jul 5, 2020 at 20:45 -
@IanCampbell thanks - it's a work in progress. It's good at extracting text and grouping into words and paragraphs. I haven't figured out a consistent way of getting it to reconstruct tables though. Please consider forking it and having a look. Jul 5, 2020 at 20:49
-
Thanks, I have encountered many issues even trying to install your package. Jul 5, 2020 at 20:55
-
@morouzian see my update for an alternative which is on CRAN and easier to install Jul 5, 2020 at 21:14
The quickest way to do this one time is to select the data columns (but not the column headings). On page 4 of your pdf that is from 1038 in the upper left to 100.00 in the bottom right. Copy that selection to the clipboard:
dta <- read.table(pipe("pbpaste")) # For Mac Os
# For Windows use dta <- read.table("clipboard")
dta <- dta[, 1:2] # Discard the last three columns
colnames(dta) <- c("ScaleScore", "Freq")
head(dta)
# ScaleScore Freq
# 1 1038 142
# 2 1171 15
# 3 1250 78
# 4 1299 269
# 5 1335 840
# 6 1364 1690
If you want to do this repeatedly, you will have to install package pdftools
.
-
Try pasting into a text file using the R script editor or Notepad to make sure you have the data in the clipboard. If not, try to copy again.– dcarlsonJul 5, 2020 at 20:30
-
Sure, but when I copy that content and paste onto a Notepad file,
read.table("clipboard")
gives me one column with everything meshed together? Jul 5, 2020 at 20:37 -
You can wrap the single vector with
x <- data.frame(matrix(unname(unlist(dta)), length(dta)/5, 5, byrow=TRUE)[, 1:2])
. Then x will be your first two columns.– dcarlsonJul 5, 2020 at 21:06