HTTP requests & JSON

Class outline:

• Review
• HTTP basics
• Making GET requests
• Parsing JSON responses
• Handling HTTP exceptions

Types of errors

These are common to all programming languages:

• Syntax errors
• Logic errors
• Runtime errors

Common syntax errors

• Extra or missing parenthesis
• Missing colon at the end of an if, while, def statements, etc.
• Missing/unmatched quotes
• Incorrect/misaligned indents

To fix a syntax error, read the message carefully and go through your code with a critical eye. 👁
...and lint your code!

Linting

To avoid syntax errors, use a linting tool to check your code as you write it. Many IDEs have a Python linter built in or available as an extension (ex: VSCode Python linting).

You can also run Pylint on your code

                    # import pylint
                    # Already installed in Replit - no import needed
                    def sum(a, b):
                        total = a + b
                    return total
                    

In Replit, switch to the Shell tab and type pylint main.py and press Enter.
When prompted to select a version, press Enter again.

Logic errors

A program has a logic error if it does not behave as expected. Typically discovered via failing tests or bug reports from users.

Spot the logic error:

                    # Which medal goes to the first place winner?
                    medals = ["gold", "silver", "bronze"]
                    first_place = medals[1]
                    print(first_place)
                    

To avoid the wrath of angry users due to logic errors, write tests!

Testing

There are several Python testing frameworks. Pytest is a good one to start with.

                    import pytest

                    medals = ["gold", "silver", "bronze"]
                    def get_medal(place):
                        return medals[place]

                    def test_get_medal():
                        assert get_medal(1) == "gold"
                    

In Replit, click Run to install Pytest, then switch to the Shell tab and type pytest main.py and press Enter.

Runtime errors

Even with testing and linting, errors can slip through!

A runtime error happens while a program is running, often halting the execution of the program.

Spot the runtime error:

                    def div_numbers(dividend, divisor):
                        return dividend/divisor

                    quot1 = div_numbers(10, 2)
                    quot2 = div_numbers(10, 1)
                    quot3 = div_numbers(10, 0)  # Cannot divide by 0!
                    quot4 = div_numbers(10, -1)
                    

To prevent runtime errors, code defensively and write tests for all edge cases.

Tracebacks

When there's a runtime error in your code, you'll see a traceback in the console.

                        def div_numbers(dividend, divisor):
                            return dividend/divisor

                        quot1 = div_numbers(10, 2)
                        quot2 = div_numbers(10, 1)
                        quot3 = div_numbers(10, 0)
                        quot4 = div_numbers(10, -1)
                    

                    Traceback (most recent call last):
                        File "main.py", line 14, in <module>
                            quot3 = div_numbers(10, 0)
                        File "main.py", line 10, in div_numbers
                            return dividend/divisor
                    ZeroDivisionError: division by zero
                    

Exceptions

Python raises an exception whenever a runtime error occurs. An exception is a mechanism in many languages used to declare and respond to "exceptional" conditions.

How an exception is reported:

                    >>> 10/0
                    Traceback (most recent call last):
                        File "<stdin>", line 1, in 
                    ZeroDivisionError: division by zero
                    

If an exception is not handled, the program stops executing immediately.

The try statement

To handle an exception (keep the program running), use a try statement.

                    try:
                        <code you want to run if all goes well>
                    except:
                        <code you want to run if this particular exception happens>
                       ...
                    

The <try statement> is executed first. If and exception is raised with the <try statement> executes, the <except statement> is executed.

Try statement example

                    try:
                        quot = 10/0
                    except:
                        print("An error happened")
                        quot = 0
                    

Getting error details

To get the error message that Python returned, we need to catch the Exception and assign it to a variable.

                    try:
                        quot = 10/0
                    except Exception as e:
                        print('Handling an error:', e)
                        quot = 0
                    

Default Exception

All exception types are derived from the parent class Exception.

• Exception can be used as a wildcard that catches (almost) everything
• It can be used on its own, or as a last resort to catch exceptions that you haven't caught with a more specific type

Types of exceptions

A few exception types and examples of buggy code:

See full list in the exceptions docs. Note: Third party Python packages often include their own custom exception types!

Catching a specific exception type

In some cases, we need to catch and handle specific exceptions. Multiple except statements that catch different exception types are allowed. It's a best practice to catch the default Exception, in addition to specific types.

                    try:
                        quot = 10/0
                    except ZeroDivisionError as e:
                        print('Handling zero division exception:', e)
                        quot = 0
                    except Exception as e:
                        print('Something else went wrong:', e)
                    

Try inside a function

                    def div_numbers(dividend, divisor):
                        try:
                            quotient = dividend/divisor
                        except ZeroDivisionError as e:
                            print('Handling zero division exception:', e)
                            quotient = 0
                        except Exception as e:
                            print('Something else went wrong:', e)
                            quotient = None
                        return quotient

                    div_numbers(10, 2)
                    div_numbers(10, 0)
                    div_numbers(10, 'foo')
                    

Depending on the case, we may want to place the return statement inside the try statement, so that there is only a return value if the code succeeds.

System Exit

To stop the program if a certain exception is encountered, move the return statement inside the try statement and raise a SystemExit() inside the except statement.

                    def div_numbers(dividend, divisor):
                        try:
                            quotient = dividend/divisor
                        except ZeroDivisionError as e:
                            print('Encountered error:', e)
                            quotient = 0
                        except Exception as e:
                            print('Something else went wrong:', e)
                            raise SystemExit()
                        return quotient

                    div_numbers(10, 2)
                    div_numbers(10, 0)
                    div_numbers(10, -1)
                    

Raising custom exceptions

You can raise an exception and cause the program to stop, even if there's no problem that Python encounters.
This is helpful when your program requires certain values or conditions in order to work properly.

                    x = 10
                    if x > 5:
                        raise Exception("x cannot exceed 5")
                    

Exercise review:

• Class9Ex1SyntaxErrors
• Class9Ex2Pytest
• Class9Ex3TryExcept

What is HTTP?

Hypertext Transfer Protocol (HTTP) is how we send data and files around the Web

HTTP works as a request-response protocol between a client and server.

• Client could be a web browser (operated by a human) or an application (like a computer running a Python program)
• Server could be a computer with files (like HTML) stored on it, or an application serving up data through an API

HTTP requests

An HTTP request consists of:

1. Method (required) GET, PUT, POST, etc
2. Request URI (required) Full URI, including any query parameters
3. Header(s) (optional) Ex: API token)
4. Body (optional) Ex: data you want to send

                        curl # Command-line tool used to send HTTP requests
                        -X GET # Method
                        -H 'Accept: application/json' # Header
                        # Request URI
                        'https://api.open-meteo.com/v1/forecast?latitude=43.07&longitude=-89.40&hourly=temperature_2m'
                    

HTTP requests

Common request methods

Why do we care about HTTP requests?

APIs! We communicate with and integrate other software and services into our apps via APIs. Many web services offer APIs:

• Google (many APIs for different services)
• AWS (and other cloud compute providers)
• Salesforce (and other CRMs)
• Github (and other dev tools)
• Social media & email marketing platforms

Python is a great language for working with APIs!

Requests module

The Requests module is a third-party library, and is the de-facto standard for sending HTTP requests and receiving responses using Python.

On your own machine, you need to install requests before you can import it into Python files

                        pip install requests
                    

Requests module documentation

Requests module

In Replit, install requests by clicking Tools > Packages, then searching for "requests"

Making a GET request

requests.get() sends an HTTP request and returns an HTTP response object

                        import requests
                        response = requests.get("https://girldevelopit.com")
                        print(vars(response))
                    

What's in a Response?

Lots of stuff! See full list. The values we're typically concerned about are:

What's in a Response?

Let's look at some response values

                            import requests

                            response = requests.get("https://girldevelopit.com")
                            print(response.status_code) # HTTP status code (200)
                            print(response.headers) # Content-type tells us what kind of response this is (text/html)
                            print(response.text) # The full body of the response (HTML, in this case)
                        

What's an API?

An Application Programming Interface (API) is a way for 2 separate computer systems to communicate. Essentially, it's a set of rules for retrieving, creating, updating and/or deleting data in another system.

There are many different types of APIs, but on the Web we're typically talking about RESTful APIs, which allow 2 applications to exchange data securely over the Internet. RESTful APIs use URIs to identify resources.

How do APIs work?

Most Web APIs accept HTTP requests and return JSON responses (more on that in a moment!).

Beyond that, each API has it's own rules for:

• What resources are available
• Which methods are allowed (GET, PUT, POST, DELETE)
• Which headers are allowed/required
• Which query parameters are allowed/required
• What type of authentication is required (if any)

Check the documentation for the API you're interested. Here's a list of free APIs to get you started!

The NASA APOD API

NASA posts an Astronomy Picture of the Day each day, which is also available via an API. To use this API, you first need to register for a key at https://api.nasa.gov

The NASA APOD API

After registering, you should recieve a key, along with a sample request.

Query parameters

In the NASA API, your key is added as a query paramter

                        ?api_key=eIkpQYXKa6mq1J75qPMYc0Rvit3K8wNaytEJCnTn&start_date=2023-01-01
                    

Query parameters ("params") are key/value pairs that allow adding extra context to a URL.

• Query param portion of a URL begins with ?
• Each additional param is preceded with &

                        https://api.nasa.gov/planetary/apod?api_key=eIkpQYXKa6mq1J75qPMYc0Rvit3K8wNaytEJCnTn&start_date=2023-01-01
                    

Params are specific to seach site/app/API - there's no universal set of available params

The NASA APOD API

Let's run this in Python

                        import requests

                        # per NASA API docs, we need to pass the key as a query parameter
                        params = {'api_key': 'eIkpQYXKa6mq1J75qPMYc0Rvit3K8wNaytEJCnTn'}
                        response = requests.get("https://api.nasa.gov/planetary/apod", params=params)
                        print(response.status_code)
                        print(response.headers)
                    

About JSON

Javascript Object Notation (JSON) is an open standard file format that stores data in key/value pairs. It looks a lot like a Python dictionary!

Requests has a built-in .json() method for parsing JSON responses, and Python has a built-in JSON library, too.

                        import requests
                        params = {'api_key': 'eIkpQYXKa6mq1J75qPMYc0Rvit3K8wNaytEJCnTn'}
                        response = requests.get("https://api.nasa.gov/planetary/apod", params=params)
                        print(response.json())
                    

JSON example

                        {
                            'copyright': 'Mike Selby',
                            'date': '2023-02-04',
                            'explanation': 'Centered in this colorful cosmic canvas, NGC 2626 is a beautiful, bright, blue reflection nebula in the southern Milky Way. Next to an obscuring dust cloud and surrounded by reddish hydrogen emission from large H II region RCW 27 it lies within a complex of dusty molecular clouds known as the Vela Molecular Ridge. NGC 2626 is itself a cloud of interstellar dust reflecting blue light from the young hot embedded star visible within the nebula. But astronomical explorations reveal many other young stars and associated nebulae in the star-forming region. NGC 2626 is about 3,200 light-years away. At that distance this telescopic field of view would span about 30 light-years along the Vela Molecular Ridge.',
                            'hdurl': 'https://apod.nasa.gov/apod/image/2302/NGC_2626_CDK_700_II_20_Jan_2023.jpg',
                            'media_type': 'image',
                            'service_version': 'v1',
                            'title': 'NGC 2626 along the Vela Molecular Ridge',
                            'url': 'https://apod.nasa.gov/apod/image/2302/NGC_2626_CDK_700_II_20_Jan_2023_1024.jpg'
                        }
                    

Accessing JSON values

Once the response values are parsed as JSON, we can treat it like a dictionary and access individual fields with bracket notation.

                        import requests
                        params = {'api_key': 'eIkpQYXKa6mq1J75qPMYc0Rvit3K8wNaytEJCnTn'}
                        response = requests.get("https://api.nasa.gov/planetary/apod", params=params)
                        url = response['url']
                        print(url)
                    

HTTP status exceptions

Requests includes custom exception types, such as HTTP error, which covers issues like 404 not found and 403 access denied (bad credentials).

Requests does not automatically raise HTTP exceptions; we need to do that explicitly with the .raise_for_status() method and handle the exception.

Handling HTTP errors

                        import requests

                        params = {'api_key': 'eIkpQYXKa6mq1J75qPMYc0Rvit3K8wNaytEJCnTn'}

                        try:
                            response = requests.get("https://api.nasa.gov/planetary/apo", params=params)
                            response.raise_for_status()
                            url = response['url']
                            print(url)
                        except requests.exceptions.HTTPError as e:
                            print("Oh no, an error happened: ", e)
                    

Exercise: Requests

Let's play with the NASA APOD API!

Class10Ex1Requests