GETting data
Overview
Teaching: 30 min
Exercises: 15 minQuestions
How do I get data from an API using the GET method?
Is there a way to modify headers, to get a specific type of result?
Objectives
Learn how to retrieve data using the GET method
Learn how to adjust headers to get desired result
Please note: These pages are autogenerated. Some of the API-calls may fail during that process. We are figuring out what to do about it, but please excuse us for any red errors on the pages for the time being.
Using GET
The site icanhazdadjoke.com offers a wide selection of dad-jokes.
Dad jokes
a wholesome joke of the type said to be told by fathers with a punchline that is often an obvious or predictable pun or play on words and usually judged to be endearingly corny or unfunny https://www.merriam-webster.com/dictionary/dad%20joke
In addition to the website, an API is available that can be accessed using the GET method.
The GET method is a generic procedure, we need a function that actually handles the behind-the-scenes-stuff for us. The library httr have an implementation:
library(httr)
Taking a quick look at the documentation we first try GET directly:
GET("https://icanhazdadjoke.com/")
Response [https://icanhazdadjoke.com/]
Date: 2023-11-06 09:23
Status: 200
Content-Type: text/html; charset=utf-8
Size: 9.81 kB
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1, minimum-s...
<meta name="description" content="The largest collection of dad jokes on the ...
<meta name="author" content="C653 Labs" />
<meta name="keywords" content="dad,joke,funny,slack,alexa" />
<meta property="og:site_name" content="icanhazdadjoke" />
...
What is returned is the response from the server. That includes much more than what we are looking for. Notable is the “Status” part, which we are told is “200”, which is server-lingo for “everything is OK”.
And what do we get? We get a webpage. We can see that the content is DOCTYPE html. That was not really what we were looking for. HTML is not that easy to work with, and contains a lot of extranious information that we do not need.
Even if the GET method is relatively simple to work with, we need to add a bit more. Again taking a look at the documentation, it appears that we need to tell the API, that we would like a specific type of response, rather than the default html, more specifically “text/plain”.
httr has helper functions to assist us. The one we need here is accept() We now use that to tell the server, that we really want a response in just text:
result <- GET("https://icanhazdadjoke.com/", accept("text/plain"))
result
Response [https://icanhazdadjoke.com/]
Date: 2023-11-06 09:23
Status: 200
Content-Type: text/plain
Size: 59 B
We still get the response from the server, telling us that Status is 200, and everything is OK. But where is our dad-joke?
It is hidden in the content of the response. It is sent to us as binary code, so we are using the content() function, also from httr to extract it:
content(result)
No encoding supplied: defaulting to UTF-8.
[1] "A book just fell on my head. I only have my shelf to blame."
There is a little warning about the encoding of the string. But now we have a dad-joke!
What if we need to retrieve a specific joke? All the jokes has an ID, that we can use for that. If we want to find that, we need a bit more information about the joke. We can get that by specifying that we would like the result of our GET-request returned as JSON.
JSON
JSON (JavaScript Object Notation) is a format for structuring, in principle, any kind for text, structured in almost any way. It consists of pairs of strings, one denoting the name of the data we are looking at, and one containing the content of that data. Each set of data fields are encapsulated in curly braces, and a data field can have subfields, also encapsulated in curly braces. It can look like this:
{ “firstName”: “John”, “lastName”: “Smith”, “phoneNumbers”:{ “type”: “home”, “number”: “212 555-1234” }, { “type”: “office”, “number”: “646 555-4567” } }
JSON is readable for both humans and computers, but can be a bit tricky to convert to dataframes if there are a lot of nested fields.
Looking at the documentation, we see an example, which indicates that what we should tell the server that we accept, should be “application/json”. The httr library contains helper functions to assist us in manipulating the header. We use accept() that sets the accept part of the header:
result <- GET("https://icanhazdadjoke.com/", accept("application/json"))
result
Response [https://icanhazdadjoke.com/]
Date: 2023-11-06 09:23
Status: 200
Content-Type: application/json
Size: 71 B
{"id":"BI699EY08Ed","joke":"Bad at golf? Join the club.","status":200}
Again - everything is nice and 200 = OK.
We also see a truncated version of the actual joke.
Let us use the content() function to extract the content:
content(result)
$id
[1] "BI699EY08Ed"
$joke
[1] "Bad at golf? Join the club."
$status
[1] 200
This data is returned as a list, which is the R-default way of handling any kind of data. Status is repeated, and now we have an id. We can use that to extract the same joke again.
NOTE: The joke returned is chosen at random. The id used here will probably be different from what we found above.
The way to retrieve a specific joke is to GET the URL:
GET https://icanhazdadjoke.com/j/<joke_id>
Where we replace the joke_id with the specific joke we want. Remember to specify the result that we want:
GET("https://icanhazdadjoke.com/j/lGJmrrzAsc", accept("text/plain")) %>%
content()
No encoding supplied: defaulting to UTF-8.
[1] "A termite walks into a bar and asks “Is the bar tender here?”"
We can also search for words in jokes. The documentation tells us, that we should send our GET request to the URL
https://icanhazdadjoke.com/search
And in the examples we get the hint, that we should format the URL as:
https://icanhazdadjoke.com/search?term=
Dogs are always fun, let us search for dad jokes about dogs. Specify the type of result we want, pipe the response to the content() function and save it to result:
result <- GET("https://icanhazdadjoke.com/search?term=dog",
accept("application/json")) %>%
content()
result
$current_page
[1] 1
$limit
[1] 20
$next_page
[1] 1
$previous_page
[1] 1
$results
$results[[1]]
$results[[1]]$id
[1] "YvkV8xXnjyd"
$results[[1]]$joke
[1] "Why did the cowboy have a weiner dog? Somebody told him to get a long little doggy."
$results[[2]]
$results[[2]]$id
[1] "82wHlbaapzd"
$results[[2]]$joke
[1] "Me: If humans lose the ability to hear high frequency volumes as they get older, can my 4 week old son hear a dog whistle?\r\n\r\nDoctor: No, humans can never hear that high of a frequency no matter what age they are.\r\n\r\nMe: Trick question... dogs can't whistle."
$results[[3]]
$results[[3]]$id
[1] "lyk3EIBQfxc"
$results[[3]]$joke
[1] "I went to the zoo the other day, there was only one dog in it. It was a shitzu."
$results[[4]]
$results[[4]]$id
[1] "DIeaUDlbUDd"
$results[[4]]$joke
[1] "“My Dog has no nose.” “How does he smell?” “Awful”"
$results[[5]]
$results[[5]]$id
[1] "EBQfiyXD5ob"
$results[[5]]$joke
[1] "what do you call a dog that can do magic tricks? a labracadabrador"
$results[[6]]
$results[[6]]$id
[1] "obhFBljb2g"
$results[[6]]$joke
[1] "I adopted my dog from a blacksmith. As soon as we got home he made a bolt for the door."
$results[[7]]
$results[[7]]$id
[1] "89MZLmWnWvc"
$results[[7]]$joke
[1] "I can't take my dog to the pond anymore because the ducks keep attacking him. That's what I get for buying a pure bread dog."
$results[[8]]
$results[[8]]$id
[1] "GtH6E6UD5Ed"
$results[[8]]$joke
[1] "What kind of dog lives in a particle accelerator? A Fermilabrador Retriever."
$results[[9]]
$results[[9]]$id
[1] "R7UfaahVfFd"
$results[[9]]$joke
[1] "My dog used to chase people on a bike a lot. It got so bad I had to take his bike away."
$results[[10]]
$results[[10]]$id
[1] "71wsPKeF6h"
$results[[10]]$joke
[1] "What did the dog say to the two trees? Bark bark."
$results[[11]]
$results[[11]]$id
[1] "AQn3wPKeqrc"
$results[[11]]$joke
[1] "It was raining cats and dogs the other day. I almost stepped in a poodle."
$results[[12]]
$results[[12]]$id
[1] "sPRnOfiyAAd"
$results[[12]]$joke
[1] "At the boxing match, the dad got into the popcorn line and the line for hot dogs, but he wanted to stay out of the punchline."
$results[[13]]
$results[[13]]$id
[1] "Lmjqzsr49pb"
$results[[13]]$joke
[1] "What did the Zen Buddist say to the hotdog vendor? Make me one with everything."
$search_term
[1] "dog"
$status
[1] 200
$total_jokes
[1] 13
$total_pages
[1] 1
This is in JSON format. It is clear that the jokes are in the $results part of that datastructure. How can we get that to a data frame?
the content() function can treat the content of our response in different ways. If we treat it as text, the function fromJSON from the library jsonlite, can convert it to a data frame. We begin by loading the library:
library(jsonlite)
Attaching package: 'jsonlite'
The following object is masked from 'package:purrr':
flatten
GET("https://icanhazdadjoke.com/search?term=dog", accept("application/json")) %>%
content(as="text") %>%
fromJSON()
No encoding supplied: defaulting to UTF-8.
$current_page
[1] 1
$limit
[1] 20
$next_page
[1] 1
$previous_page
[1] 1
$results
id
1 YvkV8xXnjyd
2 82wHlbaapzd
3 89MZLmWnWvc
4 GtH6E6UD5Ed
5 obhFBljb2g
6 R7UfaahVfFd
7 71wsPKeF6h
8 lyk3EIBQfxc
9 DIeaUDlbUDd
10 EBQfiyXD5ob
11 AQn3wPKeqrc
12 sPRnOfiyAAd
13 Lmjqzsr49pb
joke
1 Why did the cowboy have a weiner dog? Somebody told him to get a long little doggy.
2 Me: If humans lose the ability to hear high frequency volumes as they get older, can my 4 week old son hear a dog whistle?\r\n\r\nDoctor: No, humans can never hear that high of a frequency no matter what age they are.\r\n\r\nMe: Trick question... dogs can't whistle.
3 I can't take my dog to the pond anymore because the ducks keep attacking him. That's what I get for buying a pure bread dog.
4 What kind of dog lives in a particle accelerator? A Fermilabrador Retriever.
5 I adopted my dog from a blacksmith. As soon as we got home he made a bolt for the door.
6 My dog used to chase people on a bike a lot. It got so bad I had to take his bike away.
7 What did the dog say to the two trees? Bark bark.
8 I went to the zoo the other day, there was only one dog in it. It was a shitzu.
9 “My Dog has no nose.” “How does he smell?” “Awful”
10 what do you call a dog that can do magic tricks? a labracadabrador
11 It was raining cats and dogs the other day. I almost stepped in a poodle.
12 At the boxing match, the dad got into the popcorn line and the line for hot dogs, but he wanted to stay out of the punchline.
13 What did the Zen Buddist say to the hotdog vendor? Make me one with everything.
$search_term
[1] "dog"
$status
[1] 200
$total_jokes
[1] 13
$total_pages
[1] 1
We have now seen how to send a request to an API, with search terms embedded in the URL.
We have seen how to add an argument to the GET function, that specifies the type of result we would like, effectively by adding something to the header of our request.
And we have seen how to extract the results, and get them into a dataframe.
Next, we are going to take a look on how we get results using the POST method, on an API that provides more factual and serious, but not so funny data.
Exercise
Request dad jokes about cats
Solution
GET("https://icanhazdadjoke.com/search?term=dog", accept("application/json")) %>% content(as="text") %>% fromJSON()No encoding supplied: defaulting to UTF-8.$current_page [1] 1 $limit [1] 20 $next_page [1] 1 $previous_page [1] 1 $results id 1 YvkV8xXnjyd 2 82wHlbaapzd 3 EBQfiyXD5ob 4 GtH6E6UD5Ed 5 obhFBljb2g 6 89MZLmWnWvc 7 R7UfaahVfFd 8 71wsPKeF6h 9 lyk3EIBQfxc 10 DIeaUDlbUDd 11 sPRnOfiyAAd 12 AQn3wPKeqrc 13 Lmjqzsr49pb joke 1 Why did the cowboy have a weiner dog? Somebody told him to get a long little doggy. 2 Me: If humans lose the ability to hear high frequency volumes as they get older, can my 4 week old son hear a dog whistle?\r\n\r\nDoctor: No, humans can never hear that high of a frequency no matter what age they are.\r\n\r\nMe: Trick question... dogs can't whistle. 3 what do you call a dog that can do magic tricks? a labracadabrador 4 What kind of dog lives in a particle accelerator? A Fermilabrador Retriever. 5 I adopted my dog from a blacksmith. As soon as we got home he made a bolt for the door. 6 I can't take my dog to the pond anymore because the ducks keep attacking him. That's what I get for buying a pure bread dog. 7 My dog used to chase people on a bike a lot. It got so bad I had to take his bike away. 8 What did the dog say to the two trees? Bark bark. 9 I went to the zoo the other day, there was only one dog in it. It was a shitzu. 10 “My Dog has no nose.” “How does he smell?” “Awful” 11 At the boxing match, the dad got into the popcorn line and the line for hot dogs, but he wanted to stay out of the punchline. 12 It was raining cats and dogs the other day. I almost stepped in a poodle. 13 What did the Zen Buddist say to the hotdog vendor? Make me one with everything. $search_term [1] "dog" $status [1] 200 $total_jokes [1] 13 $total_pages [1] 1
Key Points
200 is the internet code for everything is OK
GET requests can be adjusted to specify desired result
Dad jokes are not really that good.