Retrieves content from any public website. Please refer to Configuration Rules for more details on the configuration of the data extraction rules.
Application
Website Automation
Inputs (what you have)
Name | Description | Data Type | Required? | Example |
Website link | Link to the website you want to extract content from. | Link | Yes | |
Data extraction rules | Instructions for what data should be extracted from the site. See the documentation for configuration details. | Text (Long) | Yes | { "title" : "h1"} |
Javascript execution | Javascript to be executed before executing the data extraction rules | Text (Long) | No |
|
Javascript rendering? | Should the Javascript run on the page being scraped. This can be useful for scraping a Single Page Application built with frameworks such as React.js, Angular.js, JQuery or Vue. This can improve the speed of this action | True/False | No |
|
Block images and CSS? | This parameter is unnecessary if JavaScript rendering is disabled. This can improve the speed of this action | True/False | No |
|
Forward headers | Specific JSON headers to forward to the website that you want to scrape | Text (Long) | No |
|
Outputs (what you get)
Name | Description | Data Type | Example |
Extracted content | JSON result of the data extracted from the website | Text (Long) | {"title": "The Wrk Blog"} |
Unsuccessful message | If unsuccessful, a message stating what went wrong | Text (Long) |
|
Note: The "Fields to Capture" input feature can be used to create new outputs for the Wrk Action with the value of the corresponding key name in the first-level JSON output of the Wrk Action.
Example: If the "Response Body" output is equal to {"title": "The Wrk Blog"} Then you are able to add a "Fields to capture" input with the name "title" that will add an output to the Wrk Action called "title" which will have the value "The Wrk Blog"
Outcomes
Name | Description |
Success | This status is selected in the event of that the API call returns with a status between 200-399 |
Unsuccessful | This status is selected in the event of that the API call returns with a status of 400+ |
Impossible to Complete | This status is selected in the event of the following scenario:
|
Configuration Rules
Regarding the configuration of the "Retrieve data from a website" Wrk Action, we recommend a helpful tool to simplify the process of setting up data extraction rules. The tool is called "jQuery Unique Selector," and you can access it by installing the Chrome extension from this link: jQuery Unique Selector Chrome Extension.
Here's how to use it:
1. After installing the extension, click on the magnifying glass icon() in your browser. This will activate the selector mode.
2. Select the specific item you want to extract from the website by clicking on it.
3. Copy the "Selected Element Selector" that the extension provides, and use it in your data extraction rules for the "Retrieve data from a website" action. Repeat this process for all the content you need to retrieve from the website.
4. To disable the extension, simply click the magnifying glass icon again.
This tool should help you with most data extraction tasks for the majority of sites. However, please keep in mind that some situations may require additional configuration. If you come across any such scenarios, don't hesitate to reach out for further assistance.
Description | Code Example |
Simple rule to extract an element's h1 text content using a CSS selector. |
|
Simple rule to extract an element's subtitle text content using an ID. |
|
Rule to extract an HTML attribute, such as the href from a link. |
|
Complex rule with a selector, specifying the desired output format. |
|
Rule to extract the HTML content of an element. |
|
Rule to extract an attribute by using the "@" prefix in the output field. |
|
Rule to extract data from a table and format it as a JSON object. |
|
Rule to extract data from a table and format it as an array. |
|
Simple syntax for extracting text content and attributes without specifying output or type. |
|
Rule to extract and format information from a table using JSON representation. |
|
Rule to extract and format information from a table using array representation. |
|
Rule to extract the first matching element for a selector. |
|
Rule to extract all matching elements for a selector. |
|
Rule for cleaning the extracted text content by default. |
|
Rule to extract content without cleaning, preserving whitespace and special characters. |
|
Nested extraction rule to gather detailed information from multiple items. |
|
Rule to extract all links from a page, returning an array of href attributes. |
|
Rule to extract text and href for each link, providing a more detailed structure. |
|
Rule to extract all textual content from a page body. |
|
Rule to extract all email addresses from a page using mailto links. |
|