Internet Automation With Selenium And Python

Hello everybody! 👋 I lately moved to Redmond, WA, and tried to get into some sports activities that might maintain me lively and shifting. After wanting round for a bit I noticed that half the individuals right here play badminton so it was a simple choice for me to join an area badminton academy. Nevertheless, after signing up I noticed that a lot of the courts have been already booked for prime-time (between 4-8 pm). I came upon that the reserving for the courts opens up on Saturday at round 10 am and the great timeslots for the subsequent complete week are booked within the first 15-20 minutes. I did what any sane programmer would do and began enthusiastic about the way to automate the entire reserving course of. This text goes to stroll you thru the guide reserving course of and clarify how I automated it.

tldr: you may obtain the supply for the bot from GitHub

Handbook reserving course of

It’s all the time vital to completely discover the guide workflow earlier than you try to automate something. We will likely be automating the Northwest Badminton Academy reserving system. That is their reservations web page (hyperlink).

This web page lists all of the courtroom reservation guidelines. It will be important for us to maintain these in thoughts as they’ll play a job in how we automate the bookings. The vital guidelines are:

1. We might solely guide 1 slot every day
2. We might guide solely 3 slots throughout weekdays (Mon-Fri)
3. We might guide 1 slot every day on Saturday and Sunday

On the backside of the web page, additionally they hyperlink to the reservation web page with a full calendar (hyperlink). When you click on on that you’ll be redirected to this web page:

Curiously, the reserving shouldn’t be hosted by NWBA themselves. They outsource it to zen planner. So primarily, we will likely be automating zen planner bookings. This additionally implies that the ultimate automation ought to work for many different locations as properly that use zen planner.

When you go forward and log in, you may go to the calendar web page (hyperlink):

The calendar exhibits which timeslots can be found, you may click on on any time which has a couple of slots open and it’ll take you to the timeslot reservation web page:

From right here you may click on on “Reserve” and the web page ought to refresh and let you know that your reservation was profitable.

And whereas following the reserving guidelines, we are able to guide a complete of 5 slots every week. 3 of them through the week and a pair of on weekends.

Now that we all know the guide reserving course of, let’s work out the way to automate this.

Getting the instruments prepared

We will likely be utilizing Python 3.9 for this challenge. We will likely be counting on Selenium for the automation. Let’s begin by making a model new folder, making a digital setting in it, and putting in selenium.

$ mkdir booking_bot
$ cd booking_bot
$ python -m venv venv
$ supply venv/bin/activate
$ pip set up selenium

Relying on which working system you’re utilizing, you’ll have to obtain the respective Chrome webdriver as properly. We will likely be utilizing Chrome driver with selenium. You possibly can go to the official Chrome driver web site and obtain the newest steady launch to your working system. You’ll find yourself with a zipper file and the zip file will include a binary known as “chromedriver”. Put this binary within the booking_bot folder.

Now we are able to go forward and begin with the precise coding.

Getting began with coding

I usually open up the Python REPL and in addition maintain an app.py file open within the background. I take a look at out new code within the REPL in an interactive vogue after which copy the code to the app.py file. This makes the iteration loop very fast and I don’t should constantly re-run the app.py code with every small change. We will likely be doing the identical right here. Create an app.py file within the booking_bot folder after which run Python in a terminal.

Let’s begin by importing selenium, firing up a Chrome occasion, and opening the NWBA login web page in it:

from selenium import webdriver

url = "https://northwestbadmintonacademy.websites.zenplanner.com/login.cfm"

browser = webdriver.Chrome(executable_path="./chromedriver")
browser.get(url)

The executable path could be totally different based mostly on which folder you’re operating Python within the terminal from and the place your chromedriver executable is saved. Modify the trail accordingly.

If all the pieces goes properly, you must now have a Chrome window open that claims “Chrome is being managed by automated take a look at software program”.

Automating login

The way in which automation with selenium works is that we have to inform selenium which HTML tags we wish to work together with and what will we wish to do with them. Automating the login entails telling Selenium which fields on the web page are the username and password fields and what worth will we wish to cross on to them after which which button is the submit button.

We may also help Selenium discover the related tags on the web page in a number of methods. We will use any of the next strategies:

• find_element_by_id
• find_element_by_name
• find_element_by_xpath
• find_element_by_link_text
• find_element_by_partial_link_text
• find_element_by_tag_name
• find_element_by_class_name
• find_element_by_css_selector

You will discover full documentation for every of those strategies over right here.

At this level, we are able to open up developer instruments in Chrome and examine the enter subject and see which methodology would possibly swimsuit our wants the most effective.

Primarily based on the HTML code for the enter fields, it looks as if we are able to simply use the title attribute of the enter tag to uniquely determine it.

user_block = browser.find_element("title", "username")
pw_block = browser.find_element_by_name("password")

We will cross in values for these fields utilizing the send_keys methodology:

user_block.send_keys("your_email")
pw_block.send_keys("your_password")

Now we have to uniquely determine the “Log In” button. I’ll present you the way to use my favourite methodology to try this. We will likely be utilizing XPath. They’ve by no means failed me and they’re versatile sufficient for use in every single place. XPath is a language used for finding nodes in an XML doc and it really works equally properly in HTML as properly. There’s a little little bit of syntax that that you must be taught however more often than not a fast Google seek for a selected use-case is sufficient.

The “Log In” button has the sort SUBMIT and we will likely be finding it based mostly on the sort. The code for that is:

submit_button = browser.find_element_by_xpath("//enter[@type="SUBMIT"]")

We’re utilizing find_element_by_xpath methodology right here as in comparison with find_elements_by_xpath. This solely returns one aspect. As for the XPath, // tells selenium to seek out the enter tag regardless of how deeply nested it’s within the doc. The kind tells it to seek out solely these enter tags which have a kind of SUBMIT.

Clicking the submit button is now as simple as:

submit_button.click on()

If all the pieces goes properly, this may redirect us to the Profile view.

Automating calendar interplay

We have to work out the way to get from the profile view to the Calendar web page. Fortunately there’s a helpful Calendar button within the sidebar.

We will find this button in a number of methods utilizing XPath. I’ll reveal two of them right here simply to point out you the ability of XPath. The primary one is:

calendar_btn = browser.find_element_by_xpath("//td[@id='idNavigation']//li[2]/a")

• //td[@id='idNavigation'] returns the td tags with the id of idNavigation
• //li[2] selects the second li nested beneath the td tag (counting begins from 1 in XPath)
• /a selects the direct a little one of li

The second methodology is:

calendar_btn = browser.find_element_by_xpath("//a[text()='Calendar']")

This selects the a tag within the HTML doc that has the textual content Calendar.

You should utilize whichever one you favor and add the follow-up code for clicking the button:

calendar_btn.click on()

This could take us to the Calendar web page.

This solely exhibits one date however we wish to navigate to whichever date we wish. There are once more two methods to do it. We will both click on the arrow icons proper subsequent to the date or we are able to work out the URL sample for dates and recreate that ourselves. Only for a little bit little bit of problem, I’ll go along with the latter possibility.

Go forward, copy the present URL, navigate to a special date and examine the 2 URLs and see what modified.

The preliminary URL was:

https://northwestbadmintonacademy.websites.zenplanner.com/calendar.cfm?calendarType=PERSON:6C482159-B1D5-47E0-8427-CCCF2EC1DAC9

The brand new URL is:

https://northwestbadmintonacademy.websites.zenplanner.com/calendar.cfm?DATE=2021-10-25&calendarType=PERSON:6C482159-B1D5-47E0-8427-CCCF2EC1DAC9&VIEW=LIST&PERSONID=6C482159-B1D5-47E0-8427-CCCF2EC1DAC9

Looks as if there are two dynamic components of the URL. The primary one is the date and the second is a PERSON identifier. I’m wondering if we are able to get the PERSON identifier from the HTML doc. The identifier is part of the preliminary URL as properly so it looks as if the Calendar button already has it. We will extract the identifier very simple:

user_id = calendar_btn.get_attribute('href').cut up('=')[-1].cut up(':')[-1]

We had already situated the calendar button. We simply wanted the HREF attribute of that tag and Selenium makes it tremendous simple to extract attributes from tags.

We don’t want the entire HREF so we cut up it at =:

['https://northwestbadmintonacademy.sites.zenplanner.com/calendar.cfm?calendarType', 'PERSON:6C482159-B1D5-47E0-8427-CCCF2EC1DAC9']

We then take the latter half and cut up it at : and take the final aspect from the returning record:

['PERSON', '6C482159-B1D5-47E0-8427-CCCF2EC1DAC9']

Now we are able to recreate the precise date URL ourselves:

query_date = "2021-10-24"
calendar_date_link = f"https://northwestbadmintonacademy.websites.zenplanner.com/calendar.cfm?DATE={query_date}&calendarType=PERSON:{user_id}&VIEW=LIST&PERSONID={user_id}"

Whereas we’re at it, let’s create a separate methodology to return the subsequent 7 dates:

import datetime

def next_7_dates():
    at present = datetime.datetime.at present()
    date_list = []
    for x in vary(0,7):
        new_date = at present + datetime.timedelta(days=x)
        date_list.append(new_date.strftime('%Y-%m-%d'))
    return date_list

We use the datetime library to get at present’s date after which use timedelta so as to add extra days to it and at last use strftime to solely extract the 12 months (%Y), month (%m), and day (%d) from it.

Let’s take a look at the calendar itself now. We will click on on any inexperienced date that has a couple of spots open and go to the ultimate reserving web page:

I usually desire to play at 6 pm so we are able to add some logic the place our automation bot seems to be on the accessible instances and figures out if our favourite time is out there or not. If it isn’t accessible it should then go to the subsequent web page. That is easy sufficient to code:

booking_link = browser.find_element_by_xpath("//div[text()='6:00 PM']")
full="sessionFull" in booking_link.get_attribute('class')
if not full:
    booking_link.click on()

We’re making use of the constructing blocks that we have now already realized about. We find the div tag that has our required time as textual content. We then examine if there are any slots accessible through the existence of sessionFull class on the div tag. If the timeslot shouldn’t be full, we click on on the reserving hyperlink.

Automating the ultimate reserving

After clicking on one of many timeslot hyperlinks, we must always now be redirected to the ultimate reserving web page. From right here we simply must click on on the Reserve button after which confirm that we bought booked.

We will click on on the Reservation button utilizing the next code:

browser.find_element_by_xpath("//a[text()='Reserve']").click on()

Afterward, the web page ought to refresh and you must see the next affirmation display screen:

We will examine for the existence of the Reserved div to substantiate that our reserving went by with out a hitch:

reserved = "Reserved" in browser.find_element_by_xpath("//div[@class="bold green"]").textual content

Now we have now all of the required items and we simply must put them in our app.py file and provides them some construction.

Remaining code

The ultimate code seems to be one thing like this:

import datetime
from selenium import webdriver
from selenium.webdriver.help import expected_conditions
from selenium.webdriver.help.ui import WebDriverWait
from selenium.webdriver.widespread.by import By

def next_7_dates():
    at present = datetime.datetime.at present()
    date_list = []
    for x in vary(0,7):
        new_date = at present + datetime.timedelta(days=x)
        date_list.append(new_date.strftime('%Y-%m-%d'))
    return date_list

def login(browser):
    print("[+] Logging in.")
    browser.find_element("title", "username").send_keys("your_email")
    browser.find_element("title", "password").send_keys("your_password")
    browser.find_element("xpath", "//enter[@type="SUBMIT"]").click on()

def reserve_time(browser, favorite_times):
    for fav_time in favorite_times:
        booking_link = browser.find_element("xpath", f"//div[text()='{fav_time}']")
        full="sessionFull" in booking_link.get_attribute('class')
        if not full:
            booking_link.click on()
        else:
            proceed
        browser.find_element("xpath", "//a[text()='Reserve']").click on()
        reserved = "Reserved" in browser.find_element("xpath", "//div[@class="bold green"]").textual content
        if reserved:
            return True, fav_time
    return False, None

def principal():
    browser = webdriver.Chrome(executable_path=r'/usr/native/bin/chromedriver')
    url = "https://northwestbadmintonacademy.websites.zenplanner.com/login.cfm"
    browser.get(url)

    login(browser)

    timeout_secs = 20
    calendar_btn = WebDriverWait(browser, timeout_secs)
                        .till(expected_conditions.presence_of_element_located((By.XPATH, "//td[@id='idNavigation']//li[2]//a")))
    user_id = calendar_btn.get_attribute('href').cut up('=')[-1].cut up(':')[-1]
    calendar_btn.click on()

    favorite_times = ["5:00 PM", "6:00 PM"]
    booked_details = []
    for rely, date in enumerate(next_7_dates()):
        if len(booked_details) == 3 and rely <= 5:
            print(f"[+] Already booked 3 weekdays. Skipping {date}")
            proceed
        print(f"[+] Making an attempt to search for timeslots on {date}")
        calendar_date_link = (f"https://northwestbadmintonacademy.websites.zenplanner.com/calendar.cfm?"
            f"DATE={date}&calendarType=PERSON:{user_id}&VIEW=LIST&PERSONID={user_id}")
        browser.get(calendar_date_link)
        reserved, reservation_time = reserve_time(browser, favorite_times)
        if reserved:
            booked_details.append((date, reservation_time))

    print("[+] I used to be capable of efficiently reserve the next date/instances:")
    for date, reservation_time in booked_details:
        print(f"t{date}: {reservation_time}")

if __name__ == "__main__":
    principal()

The code is split into 4 capabilities and most of them are self-explanatory. Nevertheless, there are some things in right here that we didn’t focus on above. Whereas operating the ultimate app.py, I came upon that find_element_by_* strategies are deprecated, and as an alternative I ought to use the find_element methodology.

Furthermore, I used to be encountering the StaleElementReferenceException whereas finding the calendar button within the HTML. This StackOverflow reply provides an excellent clarification of this exception and in addition provides an answer. What was taking place is that we have been finding the calendar button whereas the browser was nonetheless ending the logging in course of and as quickly as we tried to entry the calendar button, it had turn into stale. To be able to overcome this, we have now to attend a bit for the calendar aspect to turn into current on the web page after the logging-in course of has already began. This manner Selenium waits for the method to complete and makes certain the calendar button doesn’t turn into stale earlier than we attempt to entry it.

The code for that is:

from selenium.webdriver.help import expected_conditions
from selenium.webdriver.help.ui import WebDriverWait
from selenium.webdriver.widespread.by import By

# ....

calendar_btn = WebDriverWait(browser, timeout_secs)
                        .till(expected_conditions.presence_of_element_located((By.XPATH, "//td[@id='idNavigation']//li[2]//a")))

Selenium waits till the anticipated situation turns into true earlier than it continues the execution of the remainder of the code.

We may have overcome this challenge by including an specific time.sleep(10) as properly however the issue with that’s the aspect would possibly turn into accessible prior to 10 seconds however we’d not be capable of do something earlier than the entire 10 seconds have elapsed.

As for the code circulation, we begin execution with the principle methodology. It creates the browser window and logs us in. It then loops by the subsequent seven days and for every day it tries to guide our favourite time. After reserving one timeslot for a day it strikes on to the subsequent day and skips every other timeslots accessible for a similar day. Earlier than shifting on to the subsequent day, it additionally verifies that we haven’t booked greater than 3 weekdays earlier than shifting on to the subsequent weekday. Lastly, it prints the slots that it was capable of guide.

You would possibly want to alter two particulars on this code. One is the username and password values and the opposite is the favorite_times record.

Now you may run this script on Saturday morning as quickly because the bookings open up and you must hopefully be capable of snag a couple of good timeslots.

Conclusion

This was a enjoyable challenge to discover a little bit bit about Selenium and I’m hoping you realized one thing new from it. I like leaving individuals with some tips about the way to additional develop a tutorial challenge. There are such a lot of methods you may enhance this and some issues that come to thoughts are:

1. Make Chrome run in headless mode in order that no window opens up on the display screen
2. Make it run robotically on Saturday so that you just don’t even should get up (look into crontab or scheduled GitHub Actions)
3. Add higher error dealing with and possibly spawn a number of Selenium processes so as to examine a number of days in parallel

The chances are limitless! I hope you loved this tutorial. You possibly can obtain the entire supply for the bot from GitHub. In the event you appreciated this text, you may learn comparable enjoyable stuff over at my weblog. Goodbye!