Collecting RSS Feeds and Crawling Websites — Building an n8n Pipeline, Part 2

Hello! Part 2 covers the first stage of the pipeline: scheduling automatic runs and collecting data.

It's time to build the part of the bot that retrieves the latest news from websites at a set time each morning. We'll start with RSS and then work through a practical example of crawling a site that doesn't offer a feed.

The node acts like an alarm clock that starts the pipeline at the times you choose. In computing, a job that repeats on a fixed schedule is called a job.

Find the Schedule Trigger node using n8n's search box on the left and add it to the canvas.

Open the settings, select Days as the interval, and set the time to 8 a.m. Since we set the Docker time zone to Asia/Seoul, the first step will run at exactly 8 a.m. Korea time every morning.

is the most reliable way to collect data from a website that offers it. A feed is a standard format in which the site publishes article titles and links for readers and other tools to consume.

Search for and connect it after the Schedule Trigger node. In the node settings, enter the RSS feed address of the technology news blog you want to follow in the URL field. For this example, use https://feeds.arstechnica.com/arstechnica/technology-lab.

When you run the node, the collected data appears as JSON on the right. The feed structure stays the same even if the website's design changes, which makes RSS useful for building a reliable data collection pipeline locally.

Some sites don't provide RSS feeds. To collect data from those sites, we can analyze their HTML and pull out the information we need, a process called web crawling or scraping.

We'll use , a popular news site among developers worldwide, for the hands-on example. First, connect an node to the Schedule Trigger.

Set Method to GET and enter https://news.ycombinator.com/ in the URL field. When you run the node, it fetches the full HTML source of the Hacker News page, just as a browser does when you visit it.

The HTML document forms a data structure. Starting from the root tag, child tags branch out into a hierarchy.

To extract just the article titles and links from that tree, we need , which describe how to find the elements we want. Connect an node directly after the HTTP Request node.

On Hacker News, the article titles share the same path through the tags, span.titleline > a. Enter the following values in the extraction node settings to get the underlying data.

Run the node and you'll see just the requested titles and links in arrays, without the surrounding HTML. Combining HTTP Request and HTML extraction nodes gives you a straightforward way to collect data from sites without RSS feeds.

We worked through automated scheduling and data collection, the first stage of the pipeline. We also learned how to extract data from sites without RSS by navigating an HTML tree.