C# Net Scraping and Automation

I believe each programmer tries to automate a few of their duties. As soon as, considered one of my colleagues created an app that checked the cinema web site so as to guide a ticket to a Star Wars film.

In C# the are some ways to scrape an internet site or automate a circulate on an internet site. Here’s a record of attainable choices:

Selenium Webdriver

Selenium is an automation testing framework that may be additionally used to file human actions on web sites.

It’s the most well-liked alternative for web site automation.

I don’t suggest utilizing it for scraping, nevertheless it’s helpful for dynamic pages. You probably have a web page the place the knowledge is hidden behind a button click on, then Selenium may be a alternative.

Selenium provides the chance to management an actual browser. It has by default assist for Chrome, Web Explorer, Firefox, and a headless browser that doesn’t supply a visible window.

A headless browser is helpful as a result of it doesn’t want a lot RAM reminiscence.

Puppeteer Sharp

Puppeteer is just like Selenium. Which means that within the again an precise browser masses the pages.

In my view, Puppeteer is extra highly effective for the automation of the UI than Selenium. It additionally provides the chance to create PDF information primarily based on the outcomes.

Puppeteer is simpler to be taught than Selenium. Search for instance, how simple is to take a screenshot of this weblog.

utilizing var browserFetcher = new BrowserFetcher();
await browserFetcher.DownloadAsync();
await utilizing var browser = await Puppeteer.LaunchAsync(
    new LaunchOptions
    {
        Headless = false,
        ExecutablePath= @"C:Program FilesGoogleChromeApplicationchrome.exe",
        IgnoredDefaultArgs = new string[] { "--disable-extensions" }

    });
await utilizing var web page =(await browser.PagesAsync())[0];

await web page.GoToAsync("http://programmingcsharp.com");
await web page.ScreenshotAsync("screenshot.png");

Net Browser in Home windows Kinds

It is a beneficial strategy to automate browser actions and supply an enhanced view of the actions in your Home windows Type or WPF Software.

There are a number of net browser cases that can be utilized within the Home windows Kinds framework.

WebBrowser2 Management Class

Within the Home windows Kinds framework, there’s a default management that provides you an internet browser in your kind. This net browser may be managed.

You possibly can navigate to an internet site, discover the textual content packing containers through the use of a selector, fill out the shape and submit it.

personal async void button1_Click(object sender, EventArgs e)
{
    this.webView21.Supply= new Uri("https://programmingcsharp.com");
    await webView21.ExecuteScriptAsync("doc.querySelector("enter[value="Sign up"]").click on();");
}

On this manner, you may automate some actions immediately out of your kind.

I like to recommend you begin with this tutorial to get began with WebView2 in your Home windows Type Software.

CefSharp – Embedded Chromium browser

CefSharp is an open-source library and it permits you to embed a Chromium browser in your Home windows Type Software or WPF.

It’s similar to WebView2 management. It has assist for 32-bit and 64-bit CPUs.

HTML Agility Pack

HTML Agility Pack is a .NET library that may load HTML pages and parse them.

When you loaded the HTML, you should use XPATH to pick the knowledge that you simply want. There may be additionally the chance to make use of CSS selectors or different HTML attributes.

At the side of LINQ, you may traverse the doc, and flick thru the doc.

The HTML Agility Pack library provides a variety of helper strategies so as to assist you to govern the DOM.

In my view, that is the only option if you wish to scrape the Web utilizing C#.

Search for instance how simple is to learn the headers from my weblog:

HtmlWeb net = new HtmlWeb();
var htmlDoc = net.Load("https://programmingcsharp.com");
var nodes = htmlDoc.DocumentNode.SelectNodes("//physique//h2");

foreach (var node in nodes)
{
    Console.WriteLine(node.InnerText);
}

AngleSharp

AngleSharp is a library that can be utilized to parse HTML, CSS, XML, or JavaScript.

It’s similar to Html Agility Pack, which is taken into account extra well-liked.

After you load the HTML supply code, you should use LINQ on the doc object. So, it’s simple to seek out your required data.

var config = Configuration.Default.WithDefaultLoader();
var deal with = "https://programmingcsharp.com";
var context = BrowsingContext.New(config);
var doc = await context.OpenAsync(deal with);
var headers = "physique h2";
var cells = doc.QuerySelectorAll(headers);
var titles = cells.Choose(m => m.TextContent);

AngleSharp can deal with SVG and MathML components.

RestSharp

RestSharp is a REST API shopper.

REST is a protocol that many massive web sites use so as to expose their information and options.  For instance, you may get the Twitter profile avatar by calling a REST API.

Each time once you need to get some information from an internet site, search first for REST providers. If there are some public providers out there, then use them.

Many web sites supply a REST API so as to keep away from scrapers that overload the servers. An API will return solely the wanted information with out overhead like HTML and CSS.

Iron Net Scraper

Iron Net Scraper is a product that means that you can scrape web sites.

The distinction between this product and different open-source tasks is that Iron Software program has a suite of merchandise that may assist you to automate issues:

1. PDF – create, learn and edit PDF information
2. Iron OCR – Optical Character Recognition that helps a number of languages and codecs
3. Iron XL – automate Microsoft Workplace Excel
4. Iron Barcode – learn and write QR and barcodes

Conclusions about C# automation

There are numerous different instruments that you should use to automate issues. If none doesn’t suit your wants, then be happy to create your individual software.

Check out HttpClient class, it’s the primary class that may carry out HTTP requests. You possibly can obtain the supply code and use a library like AngleSharp to parse the HTML.

On the half, if you wish to create bots for some web sites, first, attempt to search if they provide a public service for his or her options. Some web sites provide you with without cost a variety of information.

In conclusion, check out totally different potentialities after which select one or a number of libraries.