GeoDCPresentation.Rpres

Parsing Public Health PDF's
========================================================
author: Collin Schwantes
date: 04 February 2019
css: custom.css

https://github.com/collinschwantes/GeoDCPublicHealthPDFs

Why Should I care? 
========================================================
 
- Ebola
- Zika
- Vaccine preventable diseases
- Lead levels

Why PDFs?
========================================================

### PDF's are EVERYWHERE in Govt. and especially in Public Health
- World Health Organization
- Minstry of Health Madagascar
- Your local public health department*
  - * willing to bet you a beer

Why Are PDF's hard to work with?
========================================================

- Designed to be human readable
- Designed to be printed
- Generally not designed as a machine-readable data store

But isn't the XML uniform and manipulatable?
========================================================
# Not really
![SoCloseDog](https://media.giphy.com/media/jBAhj7U5Zptvy/giphy.gif)

So you're just manipulating strings?
========================================================
Hopefully!

![SwimBabyturtle](https://media.giphy.com/media/SYXo0GITROlEs/giphy.gif)


What are my options?
========================================================
- PDFTools - text based pdfs
- Tabulizer - Java-based tool with R wrapper
- Tesseract - OCR without leaving R 

Why use OCR?
========================================================

- PDFs and websites often contain content in images
- Adobe OCR works pretty well for tables sometimes 
- Tesseract can provide a programtic way of extracting text 

An Example: OCR
================

![EbolaTable](https://gallery.mailchimp.com/89e5755d2cca4840b1af93176/images/62e6af37-9999-49f3-b6ef-30b56ede0459.png)

An Example: OCR
==================

![EbolaTable](https://gallery.mailchimp.com/89e5755d2cca4840b1af93176/images/62e6af37-9999-49f3-b6ef-30b56ede0459.png)
***
- Health Zones nested in provinces
- Case status nested in category "cumulative"
- Different colors
- French (bias is in training data is real)
- Sparse text

What can go wrong: OCR Tesseract
========================================================

## I TRIED
it is finnicky, the params list is a mile long
```{r, echo=FALSE}
library(tesseract)
library(dplyr)

EbolaText <- ocr(engine = tesseract("fra"),image = "https://gallery.mailchimp.com/89e5755d2cca4840b1af93176/images/62e6af37-9999-49f3-b6ef-30b56ede0459.png")

cat(substr(EbolaText,start = 1,stop = 200))
```


How to fix: OCR issues
========================================================
Read the documentation 
- [docs](https://github.com/tesseract-ocr/tesseract/wiki/ControlParams)
- [tutorial](http://www.joyofdata.de/blog/a-guide-on-ocr-with-tesseract-3-03/)
- Preprocess your image
- train a new engine with data from your region of interest

How to fix: OCR result
=========================================================

Before
```{r, echo=FALSE}
library(tesseract)
library(dplyr)

EbolaText <- ocr(engine = tesseract("fra"),image = "https://gallery.mailchimp.com/89e5755d2cca4840b1af93176/images/62e6af37-9999-49f3-b6ef-30b56ede0459.png")

cat(substr(EbolaText,start = 1,stop = 200))
```
***
After - needs improvement
```{r,echo=FALSE}
library(magick)
library(tesseract)

img <- "https://gallery.mailchimp.com/89e5755d2cca4840b1af93176/images/62e6af37-9999-49f3-b6ef-30b56ede0459.png"

imgp <- img %>%   image_read() %>%
  image_resize("2000x") %>%
  image_convert(type = 'Grayscale') %>%
  image_trim(fuzz = 40) 

#set engine parameters to look for tables

EbolaText <- ocr(engine = tesseract("fra",
                      options = list(tessedit_pageseg_mode = 'auto',
                           textord_tabfind_find_tables = '1',
                           textord_tablefind_recognize_tables = '1')),
                        image = imgp)

cat(substr(EbolaText,start = 1,stop = 200))
```


Other OCR Tricks: Cropping
=======================================================
![EbolaCrop](./examplePDFs/EbolaCropHZ.png)
***
```{r,echo=FALSE}
library(magick)
library(tesseract)

img <- "./examplePDFs/EbolaCropHZ.png"

imgp <- img %>%   image_read() %>%
  image_resize("2000x") %>%
  image_convert(type = 'Grayscale') %>%
  image_trim(fuzz = 40) 

#set engine parameters to look for tables

EbolaText <- ocr(engine = tesseract("fra",
                      options = list(tessedit_pageseg_mode = 'auto',
                           textord_tabfind_find_tables = '1',
                           textord_tablefind_recognize_tables = '1')),
                        image = imgp)

cat(EbolaText)
```


Tabulizer
========================================================

If you can get java JDK and R to connect properly it probably works great. [Troubleshooting docs](https://github.com/ropensci/tabulizer#installing-java-on-windows-with-chocolatey) from the great folks at ROpenSci

![kittens](https://media.giphy.com/media/vFKqnCdLPNOKc/giphy.gif)

PDFTools
==========================================

"Utilities based on 'libpoppler' for extracting **text**, fonts, attachments and metadata from a PDF file. Also supports high quality rendering of PDF documents into PNG, JPEG, TIFF format, or into raw bitmap vectors for further processing in R"

- great for getting text from pdf's
- can also be used for getting images embedded in PDF

PDFTools: PAHO Example
=========================================



![PAHO](./examplePDFs/Paho.png)

PDFTools: PAHO Example
======================================
*Before you start, review [regex](https://github.com/rstudio/cheatsheets/raw/master/regex.pdf) 

```{r,eval=FALSE}

library(pdftools)
library(stringr)
library(dplyr)
library(purrr)
library(httr)

PahoDF <- GET(url = "https://www.paho.org/hq/index.php?option=com_docman&view=download&category_slu=cumulative-cases-pdf-8865&alias=43296-zika-cumulative-cases-4-january-2018-296&Itemid=270&lang=en")

PDFraw <- content(x = PahoPDF,as = "raw")

writeBin(object = PDFraw, con = "./ExamplePDFs/Paho.pdf")


pathPAHO <- list.files(path = "./ExamplePDFs",pattern = "Paho.pdf",full.names = T)

pdf_text(pdf = "./examplePDFs/Paho.PDF")

```


PDFTools: PAHO Example
======================================

```{r,echo=FALSE}

library(pdftools)
library(stringr)
library(dplyr)
library(purrr)
library(httr)



pathPAHO <- list.files(path = "./ExamplePDFs",pattern = "Paho.pdf",full.names = T)

pahotext <- pdf_text(pdf = pathPAHO)[[1]]

matches <- gregexpr(pattern = "North America|Subtotal",text  = pahotext)

 str_sub(string = pahotext,start = (matches[[1]][1]+14),end = matches[[1]][length(matches[[1]])]
)


```

PDFTools: PAHO Example
======================================

```{r,echo=FALSE}

library(pdftools)
library(stringr)
library(dplyr)
library(purrr)
library(httr)



pathPAHO <- list.files(path = "./ExamplePDFs",pattern = "Paho.pdf",full.names = T)

pahotext <- pdf_text(pdf = pathPAHO)[[1]]

matches <- gregexpr(pattern = "North America|Subtotal",text  = pahotext)

tableText <- str_sub(string = pahotext,start = (matches[[1]][1]+14),end = matches[[1]][length(matches[[1]])]
)

str_split(string = tableText,pattern = "\n")[[1]][c(10,21,22,38)]


```

PDFTools: PAHO Example
======================================

```{r,echo=FALSE}

library(httr)
library(stringr)
library(dplyr)
library(pdftools)
library(geonames)

#set geonames username and host using options 
# options(geonamesUsername = "my.geoname",geonamesHost = "api.geonames.org")

PahoPDF <- GET(url = "https://www.paho.org/hq/index.php?option=com_docman&view=download&category_slug=cumulative-cases-pdf-8865&alias=43296-zika-cumulative-cases-4-january-2018-296&Itemid=270&lang=en")

PDFraw <- content(x = PahoPDF,as = "raw")

dir.create("./examplePDFs")

writeBin(object = PDFraw, con = "./ExamplePDFs/Paho.pdf")

textPAHO <- pdf_text(pdf = "./examplePDFs/Paho.PDF")

pahotext <- pdf_text(pdf = pathPAHO)[[1]]

#see note below
pahotext <- gsub(pattern = "Guadeloupe\n               ⁷,⁹",replacement = "Guadeloupe",x = pahotext)
#see note below
matches <- gregexpr(pattern = "North America|Subtotal",text  = pahotext)

tableText <- str_sub(string = pahotext,
                     start = (matches[[1]][1] + 14),
                     end = c(matches[[1]][length(matches[[1]])]-1))


tableText <- gsub(pattern = "Guadeloupe\n               ⁷,⁹",replacement = "Guadeloupe",x = tableText)

tableText <- gsub(pattern = "Saint Barthelemy\n                     ⁷",replacement = "Saint Barthelemy",x = tableText)

#split on line breaks
PAHOstr <- str_split(string = tableText,pattern = "\n")

# str(PAHOstr)

a <-  PAHOstr %>% 
  map(str_split,pattern = "     ") %>% 
  flatten()


## look for patterns
#a %>% map(length) %>% unlist %>% mean()
#a %>% keep(~length(.x) < 37)

b <- a %>% discard(~length(.x) < 37) %>% map(trimws) %>% 
  map(gsub,pattern = ",", replacement = "") 

# you try to do things the right way, 
#until its too easy to do it the wrong way

ValuePuller <- function(x) {
  ab <- list()
  
  for(i in 1:length(x)) {
    
    li <- x[i]
    
    if(nchar(li) > 0) {
      ab[i] <- li 
    } 
  }
  return(unlist(ab))
}

dataPaho <- b %>% map(ValuePuller)

PAHOtb <- do.call(rbind,dataPaho)

PAHOtb <- as_tibble(PAHOtb)

names(PAHOtb) <- c("Country", "Suspected","Confirmed", "Imported","Incidence","Deaths","ZikaCS","Pop")

PAHOtb$Country <- gsub(pattern = "[^[:alpha:]| ]",replacement = "",x = PAHOtb$Country)

PAHOtb <- PAHOtb %>% filter(Country != "Subtotal")

PAHOtb$Country <- gsub(pattern = "Virgin Islands UK",replacement = "British Virgin Islands",x = PAHOtb$Country)
PAHOtb$Country <- gsub(pattern = "Saint Martin",replacement = "MF",x = PAHOtb$Country)
PAHOtb$Country <- gsub(pattern = "Venezuela Bolivarian Republic of",replacement = "Venezuela",x = PAHOtb$Country)

PAHOtb

```

PDFTools: PAHO Example
=============

- Clean up data types
- Connect Admin areas to standard names
- Join to geopatial data (polygons)
- Display the data!

PDFTools: PAHO Example
=============
```{r, echo= FALSE}
shiny::includeHTML("./leafletMap.html")
```