A Very Fundamental Scraper/Aggregator Web site in Subsequent.js with Go Cloud Capabilities and Supabase

Wouldn’t it’s neat to have aggregated information (for a web site, each day e mail, push alert, and many others) of youngsters occasions in our surrounding space so we learn about them instantly?

— My spouse, probably salty we missed out on Bluey Reside tickets in Portland

That’s correct nerd sniping on a Saturday morning a number of weeks again after I didn’t have an excessive amount of else to do.

Up to now, I’ve been responsible of waving my fingers at concepts that “simply” mixture information from different web sites. That’s only a scraper web site, I’d assume. Hit some web sites, scrape what you want, chuck it in a database, and voila. This Saturday: OK — huge boy — however you continue to gotta do it — you assume you bought what it takes? 👀

Wanna hear this talked about with mouthblogging? Dave and I chatted about it on ShopTalk.

First — what’s going to do the scraping?

I occurred to have run throughout Colly the opposite day.

This selection tickled me in 3 ways:

1. I get to make use of a scraper expertise I’ve by no means seen or used earlier than.
2. It’s in Go, and I like alternatives to be taught extra Go, one thing I’ve been investing time in studying for work.
3. I can attempt to make it work as a Netlify perform as a result of they assist Go, which is yet one more studying alternative.

So the very very first thing I did was write the naked minimal Go code to confirm the concept was possible in any respect.

The Scrapable Web site

This web site, as a primary take a look at, fortuitously, is well scrapable.

Right here’s the code to go to the web site and pluck out the helpful info:

bundle fundamental

import (
  "fmt"

  "github.com/gocolly/colly"
)

func fundamental() {
  c := colly.NewCollector(
    colly.AllowedDomains("www.portland5.com"),
  )

  c.OnHTML(".views-row", func(e *colly.HTMLElement) {
    titleText := e.ChildText(".views-field-title .field-content a")

    url := e.ChildAttr(".views-field-title .field-content a", "href")

    date := e.ChildAttr(".date-display-single", "content material")

    venue := e.ChildText(".views-field-field-event-venue")

    fmt.Println(titleText, url, date, venue)
  })

  c.Go to("https://www.portland5.com/event-types/children")
}

Code language: Go (go)

The above code was working for me, which was encouraging!

Do web sites homeowners care if we do that? I’m certain some do, the place it’s a aggressive benefit factor for them so that you can have to go to their web site for info. Bandwidth prices and such in all probability aren’t a lot of a priority as of late, until you might be actually hammering them with requests. However within the case of this occasion scraper, I’d assume they might be glad to unfold the phrase about occasions.

The highlighted strains above are an actual showcase of the fragility of this code. That’s the identical as doc.querySelectorAll(".views-row .views-field-title .field-content a"). Any little change on the father or mother web site to the DOM and this code is toast.

Printing to plain out isn’t a very helpful format and this code typically isn’t very usable. It’s not formatted as a cloud perform and it’s not giving us the info in a format we will use.

An Occasion Sort

That is the type of information format that may be helpful to have in Go:

sort KidsEvent struct {
  ID      int    `json:"id"`
  Title   string `json:"title"`   
  URL     string `json:"url"`     
  Date    string `json:"date"`
  Venue   string `json:"venue"`
  Show bool   `json:"show"` 
}
Code language: Go (go)

As soon as we’ve got a struct like that, we will make a []KidsEvent slice of them and json.Marshal() that into JSON. JSON is nice to work with on the net, natch.

My subsequent step was to have the cloud perform do that scraping and return JSON information instantly.

Returning JSON from a Go Cloud Operate

That’s structured like this, which is able to run on Netlify, which makes use of AWS Lambda underneath the hood:

bundle fundamental

import (
  "context"
  "encoding/json"
  "log"

  "github.com/aws/aws-lambda-go/occasions"
  "github.com/aws/aws-lambda-go/lambda"
)

func handler(ctx context.Context, request occasions.APIGatewayProxyRequest) (*occasions.APIGatewayProxyResponse, error) {
  
  kidsEvents, err := doTheScraping()
  if err != nil {
    log.Deadly(err)
  }

  b, err := json.Marshal(kidsEvents)
  if err != nil {
    log.Deadly(err)
  }

  return &occasions.APIGatewayProxyResponse{
    StatusCode:      200,
    Headers:         map[string]string{"Content material-Sort": "textual content/json"},
    Physique:            string(b),
    IsBase64Encoded: false,
  }, nil
}

func fundamental() {
  lambda.Begin(handler)
}
Code language: Go (go)

As soon as this was working I used to be fairly inspired!

Getting it working was a bit difficult (as a result of I’m a newb). You may wish to see my public repo if you wish to do the identical. It’s a bit journey of creating certain you’ve received all the appropriate recordsdata in place, like go.mod and go.sum, that are produced while you run the appropriate terminal instructions and do the right go get * incantations. That is all so it builds appropriately, and Netlify can do no matter it must do to ensure they work in manufacturing.

Happily, you’ll be able to take a look at these things regionally. I’d spin them up utilizing the Netlify CLI like:

netlify features:serve
Code language: Bash (bash)

Then hit the URL within the browser to invoke it like:

localhost:9999/.netlify/features/scrape

I don’t really wish to scrape and return on each single web page view, that’s wackadoo.

It’s not very environment friendly and actually not mandatory. What I wish to do is:

1. Scrape occasionally. Say, hourly or each day.
2. Save or cache the info by some means.
3. When the web site is loaded (we’ll get to constructing that quickly), it hundreds the saved or cached information, it doesn’t all the time do a contemporary scrape.

One factor that may be excellent for that is Netlify On Demand Builders. They run as soon as, cache what they return, and solely ever return that cache till they’re particularly de-cached. That’s great, and we might use that right here… besides that Netlify doesn’t assist On Demand builders with Go.

This can be a second the place I may need gone: screw it, let’s make the cloud perform in TypeScript or JavaScript. That will open the door to utilizing On Demand builders, but in addition open the door to doing scraping with one thing like Puppeteer, which might scrape the actual DOM of a web site, not simply the primary HTML response. However in my case, I didn’t actually care, I used to be enjoying and I wished to play with Go.

So since we will’t use On Demand builders, let’s do it one other means!

Let’s hold the info in Postgres

We use Postgres at CodePen so one other alternative to follow with tech used on a much more necessary venture is sweet. The place ought to I hold my little Postgres, although? Netlify doesn’t have any official integrations with a Postgres supplier, I don’t assume, however they’re at the least pleasant with Supabase. Supabase has lengthy appealed to me, so yet one more alternative right here! Just about the entire level of Supabase is offering a Postgres with good DX round it. Bought.

I used to be assuming I used to be going to have to put in writing a least a little SQL. However no — organising and manipulating the database will be fully performed with UI controls.

However certainly writing and studying from the DB requires some SQL? No once more — they’ve a wide range of what they name “Consumer Libraries” (I believe you’d name them ORMs, typically) that enable you to join and cope with the info by means of APIs that really feel far more approachable, at the least to somebody like me. So we’ll write code extra like this than SQL:

const { error } = await supabase
  .from('international locations')
  .replace({ identify: 'Australia' })
  .eq('id', 1)
Code language: JavaScript (javascript)

Sadly, Go, but once more isn’t a first-class citizen right here. They’ve consumer libraries for JavaScript and Flutter and level you in direction of a userland Python library, however nothing for Go. Happily, a fast Google search turned up supabase-go. So we’ll be utilizing it extra like this:

row := Nation{
  ID:      5,
  Identify:    "Germany",
  Capital: "Berlin",
}

var outcomes []Nation
err := supabase.DB.From("international locations").Insert(row).Execute(&outcomes)
if err != nil {
  panic(err)
}
Code language: JavaScript (javascript)

The aim is to have it mimic the official JavaScript consumer, in order that’s good. It might really feel higher if it was official, however whattayagonnado.

Saving (and really, principally Updating) to Supabase

I can’t simply scrape the info and instantly append it to the database. What if a scraped occasion has already been saved there? In that case, we’d as nicely simply replace the report that’s already there. That takes care of the state of affairs of occasion particulars altering (just like the date or one thing) however in any other case being the identical occasion. So the plan is:

1. Scrape the info
2. Loop over all occasions
3. Examine the DB if they’re already there
4. If they’re, replace that report
5. In the event that they aren’t, insert a brand new report

bundle fundamental

import (
  "fmt"
  "os"

  "github.com/google/uuid"
  supa "github.com/nedpals/supabase-go"
)

func saveToDB(occasions []KidsEvent) error {
  supabaseUrl := os.Getenv("SUPABASE_URL")
  supabaseKey := os.Getenv("SUPABASE_API_KEY")
  supabase := supa.CreateClient(supabaseUrl, supabaseKey)

  for _, occasion := vary occasions {
    end result := KidsEvent{
      Title: "No Match",
    }

    err := supabase.DB.From("occasions").Choose("id, date, title, url, venue, show").Single().Eq("title", occasion.Title).Execute(&end result)
    if err != nil {
      fmt.Println("Error!", err)
    }

    if end result.Title == "No Match" {
      var saveResults []KidsEvent
      occasion.ID = uuid.New().String()
      err := supabase.DB.From("occasions").Insert(occasion).Execute(&saveResults)
      if err != nil {
        return err
      }
    } else {
      var updateResults []KidsEvent
      err := supabase.DB.From("occasions").Replace(occasion).Eq("title", occasion.Title).Execute(&updateResults)
      if err != nil {
        return err
      }
    }
  }

  return nil
}
Code language: Go (go)

My newb-ness is absolutely displayed right here, however at the least that is practical. Notes:

1. I’m plucking these ENV variables from Netlify. I added them by way of the Netlify dashboard.
2. The ORM places the info from the question right into a variable you give it. So the best way I’m checking if the question really discovered something is to make the struct variable have that “No Match” title and examine towards that worth after the question. Feels janky.
3. I’m checking for the individuality of the occasion by querying for the url which appears, ya know, distinctive. However the .Eq() I used to be doing would by no means discover a matching occasion, and I couldn’t determine it out. Title labored.
4. The save-or-update logic in any other case works effective, however I’m certain there’s a extra logical and succinct strategy to carry out that type of motion.
5. I’m making the ID of the occasion a UUID. I used to be so stumped at the way you have to supply an ID for a report whereas inserting it. Shouldn’t it settle for a null or no matter and auto-increment that? 🤷‍♀️

The purpose of this experiment is to scrape from a number of web sites. That’s taking place within the “ultimate” product, I simply didn’t that up on this weblog put up. I made features with the distinctive scraping code for every web site and known as all of them so as, appending to the general array. Now that I give it some thought, I ponder if a Go routine would quicken that up?

Scheduling

Netlify makes working cloud features on a schedule silly straightforward. Right here’s the related little bit of a netlify.toml file:

[functions]
listing = "features/"

[functions."scrape"]
schedule = "@hourly"
Code language: TOML, additionally INI (ini)

JavaScript cloud features benefit from an in-code means of declaring this info, which I favor, so it’s one other little jab at Go features, however oh nicely, at the least it’s attainable.

Pulling the info from the database

That is the straightforward half for now. I question for each single occasion:

var outcomes []KidsEvent
err := supabase.DB.From("occasions").Choose("*").Execute(&outcomes)
Code language: JavaScript (javascript)

Then flip that into JSON and return it, similar to I used to be doing above when it returned JSON instantly after the scrape.

This might and ought to be a bit extra difficult. For instance, I ought to filter out previous occasions. I ought to in all probability filter out the occasions with Show as false too. That was a factor the place some scraped information was bogus, and quite than write actually bespoke guidelines for the scraping, I’d flip the Boolean so I had one thing the keep away from displaying it. I did that on the entrance finish however it ought to be performed within the again.

A Web site

I figured I’d go along with Subsequent.js. One more factor we’re utilizing at CodePen however I might use extra expertise with, notably the very newest model. I figured I might do that the good means by utilizing getServerSideProps so a Node server would hit the cloud perform, get the JSON information, and be capable of render the HTML server aspect.

I spun it up in TypeScript too. Ya know, for the follow! That is just about your complete factor:

export async perform getServerSideProps() {
  const res = await fetch(
    `${course of.env.SITE_URL}/.netlify/features/get-events`
  );
  const information = await res.json();

  return { props: { information } };
}

sort kidEvent = {
  id: string;
  title: string;
  date: string;
  show: boolean;
  url: string;
  venue: string;
};

export default perform Residence(props: { information: kidEvent[] }) {
  return (
      <fundamental>
        <h1>Youngsters Occasions</h1>

        <div>
          {props.information.map((occasion) => {
            const date = new Date(occasion.date).toDateString();

            if (occasion.show === false) return null;

            return (
              <div key={occasion.id}>
                <dl>
                  <dt>Title</dt>
                  <dd>
                    <h3>
                      <a href={occasion.url}>{occasion.title}</a>
                    </h3>
                  </dd>

                  <dt>Date</dt>
                  <dd>
                    <time>{date}</time>
                  </dd>

                  <dt>Venue</dt>
                  <dd>{occasion.venue}</dd>
                </dl>
              </div>
            );
          })}
        </div>
      </fundamental>
  );
}
Code language: JavaScript (javascript)

Styling

I kinda love styling stuff in Subsequent.js. It helps CSS modules out of the field, and Sass is as straightforward as npm set up sass. So I could make recordsdata like Residence.module.scss for particular person parts, and use the scoped types. It’s a smidge heavy on CSS tooling, however I admit I discover this specific alchemy pleasing.

import types from "../types/Residence.module.scss";

export default perform Residence() {
  return(
    <fundamental className={types.fundamental}
    </fundamental> 
  );
}
Code language: JavaScript (javascript)

I additionally took the chance to make use of Open Props for the values of as a lot as I probably might. That results in this type of factor:

.card {
  padding: var(--size-5);
  background: var(--gray-1);
  colour: var(--gray-9);
  box-shadow: var(--shadow-3);
  border-radius: var(--radius-3);
}
Code language: CSS (css)

I discovered Open Props very good to make use of, however I kinda want it have been by some means “typed” in that it could assist me uncover and use the right variables and present me what the values are proper inside VS Code.

Pittered Out

The very fundamentals of this factor are working and I printed it. I’m unsure I’d even name this a minimal viable prototype because it has so many tough edges. It doesn’t cope with any date complexities like occasions that run a number of days and even expiring previous occasions. It scrapes from a number of websites, however solely certainly one of them is especially fascinating, so I haven’t even confirmed that there are sufficient locations to make this an fascinating scraping job. It doesn’t ship alerts or be useful in any means my spouse initially envisioned.

But it surely did train me some issues and I had enjoyable doing it. Perhaps one other full day, and it might be in first rate sufficient form to make use of, however my vitality for it’s gone for now. As shortly as a nerd sniping can come on, it comes off.