{"componentChunkName":"component---src-components-public-workflows-single-workflow-index-tsx","path":"/ready-made-workflow-library/find-all-the-urls-and-pages-from-a-website-to-csv-format","result":{"pageContext":{"result":{"workflow":{"id":88,"name":"Find all the URLs and pages from a website to CSV format","status":"completed","started_at":"1694440343636","frequency":"","created_at":"1694511276859","updated_at":"1694511276859","description":"Crawl entire websites to capture all their URLs in a handy CSV or Google Sheet format using this ready-made workflow template.","meta_title":null,"meta_description":"<p>Nowadays, information is gold, and the ability to gather valuable web data can be the key to success.</p>\n<p>However, gathering the required information takes a lot of time and effort, especially when one needs to collect a specific type of information, such as URLs and web pages existing on a website.</p>\n<p>Luckily, Hexomatic's workflow is here to simplify the process by automating the extraction of all URLs and pages from a website and delivering the results in CSV format for your convenience.</p>\n<p>With Hexomatic, you can crawl entire websites to capture all their URLs in a handy CSV or Google Sheet format.</p>","image":"https://storage.googleapis.com/hexomatic-automation-icons/public-workflows-illustrations/Find%20all%20the%20URLs%20and%20pages%20from%20a%20website%20to%20CSV%20format.jpg","categories":[{"id":2,"name":"Marketing"}],"steps":[{"id":477,"setting":"{\"source\":{\"source_csv\":false,\"source_txt\":false},\"automation_id\":81,\"dynamic_settings\":{\"keys\":[{\"key\":null,\"settings\":{\"limit\":100,\"proxy\":{\"type\":\"\",\"country_code\":\"\"},\"browser\":false,\"url_type\":\"internal\",\"page_type\":\"page\",\"ignore_urls\":\"\"},\"inputValue\":\"https://hexometer.com\"}]},\"output_keys_data\":[\"_source_url\",\"_crawled_url\",\"_crawler_detected_on\",\"_crawler_status\",\"_crawler_link_type\",\"_crawler_url_type\"],\"predefined_settings\":null,\"integration_settings\":null,\"output_keys_data_with_types\":\"{\\\"_source_url\\\":\\\"url\\\",\\\"_crawled_url\\\":\\\"url\\\",\\\"_crawler_detected_on\\\":\\\"url\\\",\\\"_crawler_status\\\":\\\"number\\\",\\\"_crawler_link_type\\\":\\\"text\\\",\\\"_crawler_url_type\\\":\\\"text\\\"}\"}","index":0,"created_at":"1631606440232","updated_at":null,"automation":{"id":81,"name":"Crawler","url":"Crawler","meta_title":"Website Crawler | Links Extractor Automation","meta_description":"Use our website crawler to extract pages, source URLs, and internal and external links. It shows the response code for each URL and consumes an automation credit per URL.","short_description":"Crawl any website extracting pages, source URLs, and links (internal and external)","long_description":"Crawl any website extracting pages, source URLs, and links (internal and external).\nThis automation shows the response code for each URL and consumes an automation credit per URL.\nCan be used as a starting point for other automations including the SEO meta tag, email scraper, screenshot, image or file downloader, keyword finder or audit automations. (1 automation credit per crawled URL)\n\n\n\n\n\n","long_description_details":"Our crawler automation will help you to crawl any website extracting pages, source URLs, and links (both internal and external).\n<br />This automation shows the response code for each URL and consumes an automation credit per URL.\n<br />Can be used as a starting point for other automations including the SEO meta tag, email scraper, screenshot, image or file downloader, keyword finder or audit automations.\n<br />**Different ways to use our Crawler automation:**\n<br />\n> * This automation can be used as a starting point of your Hexomatic workflow.\n<br />\n\n**How to use our Crawler automation:**\n<br />Step 1: Create a blank workflow.\nStep 2: Add the Crawler automation. After enter the source URL, the URLs to be ignored, URL types, the proxy mode and set the URL limit.\nStep 3: Click continue to run or schedule the workflow.","has_source":null,"icon_url":"https://storage.googleapis.com/hexomatic-automation-icons/crawler-automation.svg"}}],"tags":["Marketing"]},"prevPath":"/ready-made-workflow-library/check-the-url-status-of-a-list-of-urls","nextPath":"/ready-made-workflow-library/scrape-articles-from-a-list-of-urls"}}},"staticQueryHashes":["3915125517"]}