How one can Deal with Api Pagination With Powershell

On this weblog submit, I’ll undergo how I deal with API pagination with Powershell when working with completely different APIs. I’ll use the rickandmorty API to showcase what pagination is and how one can work with it, in Powershell.

Pagination is rather like when you’re purchasing on the net. Let’s say you’re searching for a brand new laptop computer. While you seek for the product on the webshop, the outcomes will often be break up out into a number of pages, solely exhibiting perhaps 20-50 merchandise per web page. The explanation for that is {that a} request to the server solely exhibiting 50 merchandise in comparison with a request containing a whole bunch of merchandise, might be rather more environment friendly and sooner.

The identical factor goes for an API. Let’s say you make a request to an API which have tens of millions of search outcomes. Now if the person solely wanted the primary 50 search outcomes, there’s completely no purpose to indicate tens of millions of outcomes to the person. Due to this fact we now have pagination, this offers the person the likelihood to solely present the primary 50 or 100 search outcomes. Now if the person would wish all of the search outcomes you would need to undergo some actions to drag every thing, that is what I’ll showcase on this blogpost.

The very first thing I at all times do when interacting with an API is to learn by way of the documentation. Often, if there’s any pagination occurring it is going to be defined. It’s going to almost certainly even be defined how one can deal with the pagination.

If we check out the Rick and Morty API documentation

First, the documentation will stroll by way of the REST API, so the endpoints which you’ll be able to gather assets from

Secondly, the documentation will clarify how the pagination works

Right here you’ll be able to see that there are 4 data factors you need to use to navigate all of the pages.

The documentation explains that there are 20 paperwork(‘search outcomes’) per web page. All the data on the documentation is self-explanatory.

Often, the way in which can see that an API has pagination is that it’s going to output an “data” dictionary containing data on what number of pages there are in whole.

How one can commute within the API pagination with Powershell

Let’s begin by querying the API with Powershell:

1
2
3
4

$url = "https://rickandmortyapi.com/api/character"
$Technique = "GET"

$request = Invoke-RestMethod -Uri $url -Technique $Technique

the output will give us two properties: ‘data’ and ‘outcomes’. The data property will give us data on the pagination, so which web page are we on, and tips on how to we get to the following web page. The outcomes property will give us all the info on the search outcomes

If we check out the information property we’ll see the next:

1
2
3
4
5
6
7

PS C:> $request.data | fl


depend : 671
pages : 34
subsequent  : https://rickandmortyapi.com/api/character?web page=2
prev  :

from this data we are able to see that there are 671 whole outcomes, we are able to see that there are a complete of 34 pages, and we are able to see that we are able to name the URL https://rickandmortyapi.com/api/character?web page=2 to entry the following web page.

Now if we check out the URL for the second web page we are able to see that we now have the base-URL:

1

https://rickandmortyapi.com/api

We have now the useful resource:

after which we now have the web page selector:

Which means if we made the very same API request as earlier than however simply added ‘?page2=’ to the top of the URL we might get the outcomes from web page two.

We are able to take a look at this by checking the outcomes from web page 1 after which from web page 2 and choose the final consequence from every name.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

$Technique = "GET"
$url1 = "https://rickandmortyapi.com/api/character?web page=1"
$url2 = "https://rickandmortyapi.com/api/character?web page=2"

$page1 = (Invoke-RestMethod -Uri $url1 -Technique $Technique).outcomes | Choose -Final 1
$page2 = (Invoke-RestMethod -Uri $url2 -Technique $Technique).outcomes | Choose -Final 1

Write-Output "Final consequence from web page 1"
Write-Output -InputObject $page1

Write-Output "Final consequence from web page 2"
Write-Output -InputObject $page2

If we have a look at the output we are able to see that the consequence from web page 1 has the id ’20’, and the consequence from web page 2 has the id ’40’, which matches that we now have 20 outcomes per web page.

Final consequence from web page 1
id       : 20
identify     : Ants in my Eyes Johnson
standing   : unknown
species  : Human
kind     : Human with ants in his eyes
gender   : Male
origin   : @{identify=unknown; url=}
location : @{identify=Interdimensional Cable; url=https://rickandmortyapi.com/api/location/6}
picture    : https://rickandmortyapi.com/api/character/avatar/20.jpeg
episode  : {https://rickandmortyapi.com/api/episode/8}
url      : https://rickandmortyapi.com/api/character/20
created  : 04/11/2017 22.34.53


Final consequence from web page 2
id       : 40
identify     : Beth's Mytholog
standing   : Lifeless
species  : Mythological Creature
kind     : Mytholog
gender   : Feminine
origin   : @{identify=Nuptia 4; url=https://rickandmortyapi.com/api/location/13}
location : @{identify=Nuptia 4; url=https://rickandmortyapi.com/api/location/13}
picture    : https://rickandmortyapi.com/api/character/avatar/40.jpeg
episode  : {https://rickandmortyapi.com/api/episode/18}
url      : https://rickandmortyapi.com/api/character/40
created  : 05/11/2017 10.02.26

How one can retrieve all of the API knowledge

Now after all there’ll at all times be a time once we want all the info supplied from the API. To do that we have to make a request per web page, gather the info from that web page and append it to an object. This may be performed pretty simple, by looping by way of all of the pages by including 1 to the top of the URL: ?web page=1, ?web page=2, ?web page=3. I might want to this as much as and with web page 34 which was the final web page in line with the ‘data’ from the API we referred to as earlier.

The essential performance for looping by way of the pages would look just like under. Right here I’m utilizing the variable $y to outline the web page quantity, which is beginning at 1 and it’ll enhance with 1 by utilizing: $y++ in each loop.

1
2
3
4
5
6
7

$Technique = "GET"
$y = 1
whereas($y -le 34){
    $url = "https://rickandmortyapi.com/api/character?web page=$y"
    $requst = Invoke-RestMethod -Uri $url -Technique $Technique
    $y++
}

Now for creating the item I’d begin by creating a brand new empty ArrayList earlier than the loop. Then I’d simply add all the outcomes from the API name into the newly created listing. One factor to recollect is that in the event you for every web page add all the outcomes into the listing, it will generate an inventory containing 34 gadgets.

Often, you’ll quite wish to have an inventory containing all 671 outcomes. Due to this fact I’d create a brand new foreach loop. This loop would run by way of every web page and add each single search consequence to the array.

the logic would look similiar to the under:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

# Defining the HTTP methodology as GET request
$Technique = "GET"

# Setting the primary web page, to web page 1
$y = 1

# Creating an empty ArrayList
$allResults = New-Object -TypeName System.Collections.ArrayList

# Whereas the web page quantity is lower than or equal to 34 (Final web page quantity)
whereas($y -le 34){

    # Producing the URL and making the API request
    $url = "https://rickandmortyapi.com/api/character?web page=$y"
    $requst = Invoke-RestMethod -Uri $url -Technique $Technique

    # Looping by way of all of the search outcomes
    foreach($r in $requst.outcomes) Out-Null
    

    # Including the web page quantity by 1
    $y++
}

And naturally, in the event you wished to retrieve the info as JSON you possibly can add the next line after the loop:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

[
{
    "id": 1,
    "name": "Rick Sanchez",
    "status": "Alive",
    "species": "Human",
    "type": "",
    "gender": "Male",
    "origin": {
    "name": "Earth (C-137)",
    "url": "https://rickandmortyapi.com/api/location/1"
    },
    "location": {
    "name": "Earth (Replacement Dimension)",
    "url": "https://rickandmortyapi.com/api/location/20"
    },
    "image": "https://rickandmortyapi.com/api/character/avatar/1.jpeg",
    "episode": [
    "https://rickandmortyapi.com/api/episode/1",
    "https://rickandmortyapi.com/api/episode/2",
    "https://rickandmortyapi.com/api/episode/3",
    "https://rickandmortyapi.com/api/episode/4",
    "https://rickandmortyapi.com/api/episode/5",
    "https://rickandmortyapi.com/api/episode/6",
    "https://rickandmortyapi.com/api/episode/7",
    "https://rickandmortyapi.com/api/episode/8",
    "https://rickandmortyapi.com/api/episode/9",
    "https://rickandmortyapi.com/api/episode/10",
    "https://rickandmortyapi.com/api/episode/11",
    "https://rickandmortyapi.com/api/episode/12",
    "https://rickandmortyapi.com/api/episode/13",

... .

Conclusion

In this blog post, I want over how to hand API pagination with Powershell, how you can easily retrieve all the information you need.

Working with APIs can be tricky sometimes, and if you are not aware of how pagination works it can be even harder to get the information you need. Unfortunately, some APIs do not have great documentation, and in some cases might not even mention pagination, so if you have some experience with it might be easier to understand what it is, and how it works.

One of the key aspects is to find out how many pages there are in total, then create a loop that alters the URL for the API call, in each loop. Then add the result from each API call to an ArrayList.

An important thing to remember is that if you are looping through each page and adding the page to the ArrayList you will get an array containing all the pages, this might be what you want, but in most cases, you would rather have an ArrayList containing all the result.

You can check if you created an ArrayList of all the pages or an ArrayList of all the results by doing the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

# ArrayList containing all the pages:
PS C:> $allResults.count

34


# ArrayList containing all the results
PS C:> $allResults.count

671