{"id":5855,"date":"2016-04-26T13:33:41","date_gmt":"2016-04-26T13:33:41","guid":{"rendered":"https:\/\/www.process.st\/templates\/scraping-broken-links-with-seo-spider-and-ahrefs\/"},"modified":"2024-02-28T17:09:37","modified_gmt":"2024-02-28T17:09:37","slug":"scraping-broken-links-with-seo-spider-and-ahrefs","status":"publish","type":"post","link":"https:\/\/www.process.st\/templates\/scraping-broken-links-with-seo-spider-and-ahrefs\/","title":{"rendered":"Scraping Broken Links with SEO Spider and Ahrefs"},"content":{"rendered":"<section id=\"initial-setup\">\n<h2> Initial setup: <\/h2>\n<div class=\"text-content\">\n<p>You're going to need:<\/p>\n<ul>\n<li><a href=\"https:\/\/www.screamingfrog.co.uk\/seo-spider\/\" target=\"_blank\" rel=\"nofollow noopener\">Screaming Frog SEO Spider<\/a> (free for searches of 500 links at a time)<\/li>\n<li><a href=\"google.com\" target=\"_blank\" rel=\"nofollow noopener\">Google<\/a> account<\/li>\n<li><a href=\"ahrefs.co\" target=\"_blank\" rel=\"nofollow noopener\">Ahrefs<\/a> account (to use <a href=\"ahrefs.com\/site-explorer\" target=\"_blank\" rel=\"nofollow noopener\">Site Explorer<\/a>)<\/li>\n<\/ul>\n<\/p>\n<\/p><\/div>\n<\/section>\n<section id=\"finding-seed-broken-links-with-google\">\n<h2> Finding seed broken links with Google: <\/h2>\n<\/section>\n<section id=\"create-a-new-google-sheet-template\">\n<h2> Create a new Google Sheet template <\/h2>\n<div class=\"image-content\">\n<figure>\n   <a href=\"https:\/\/www.process.st\/templates\/wp-content\/uploads\/2024\/02\/qk48M_m1tQdprbVxLUtDZQ.png\" alt=\"Create a new Google Sheet template\" target=\"_blank\" rel=\"noopener\"> <img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/www.process.st\/templates\/wp-content\/uploads\/2024\/02\/qk48M_m1tQdprbVxLUtDZQ.png\" \/> <\/a><br \/>\n  <\/figure>\n<\/p><\/div>\n<div class=\"text-content\">\n<p>Use this template:&nbsp;<a href=\"https:\/\/docs.google.com\/spreadsheets\/d\/1jQirKt3N1seiV-2KWHcTXI2iDywyg0U_IxggtQWe1E0\/pub?output=csv\" target=\"_blank\" rel=\"nofollow noopener\">https:\/\/docs.google.com\/spreadsheets\/d\/1jQirKt3N1seiV-2KWHcTXI2iDywyg0U_IxggtQWe1E0\/pub?output=csv<\/a><\/p>\n<p>Make a copy in your own account.<\/p>\n<p>You'll notice it has 2 tabs, one for process and one for unprocessed.<\/p>\n<p><strong>You will be working first in the unprocessed tab, dumping in the information quickly before moving it into the processed section and completing it with email address and other specifics.<\/strong><\/p>\n<p>Feel free to edit your copy of the template so it includes more branched links (instances of a single broken URL&nbsp;found linked in multiple places&nbsp;on the web).<\/p>\n<p>Attached below is the template in case you'd rather use it in Excel.<\/p>\n<\/p><\/div>\n<\/section>\n<section id=\"create-a-list-of-search-terms\">\n<h2> Create a list of search terms <\/h2>\n<div class=\"text-content\">\n<p>Your search terms should be based on the content you want to give backlinks to, so should be relevant. If you're trying to rank your productivity app, for example, you might want to look for lists of productivity tools and see if any of the tools have been discontinued. For this, you'll build a list of potential queries like this:<\/p>\n<ul>\n<li>intitle:&quot;productivity apps&quot;<\/li>\n<li>\u201cproductivity apps\u201d site:.org intitle:\u201dother resources\u201d -inurl:pdf -inurl:ppt -inurl:doc<\/li>\n<li>&quot;productivity apps&quot; resources<\/li>\n<li>&quot;productivity apps&quot;&nbsp;site:.edu -inurl:pdf -inurl:ppt -inurl:doc<\/li>\n<li>&quot;productivity apps&quot;&nbsp;site:.org -inurl:pdf -inurl:ppt -inurl:doc<\/li>\n<li>&quot;productivity apps&quot; site:.com -inurl:pdf -inurl:ppt -inurl:doc<\/li>\n<\/ul>\n<p>Read more about queries for broken link building <a href=\"http:\/\/webmeup.com\/blog\/broken-link-building.html\" target=\"_blank\" rel=\"nofollow noopener\">here<\/a>&nbsp;\u2014 they suggest queries such as:<\/p>\n<ul>\n<li><em>keyword &quot;resources&quot;<\/em><\/li>\n<li><em>keyword &quot;suggested sites&quot;<\/em><\/li>\n<li><em>keyword &quot;links&quot;<\/em><\/li>\n<li><em>keyword&nbsp; intitle:links<\/em><\/li>\n<li><em>keyword&nbsp; intitle:resources<\/em><\/li>\n<li><em>keyword&nbsp; intitle:recommended sites<\/em><\/li>\n<li><em>site:.gov keyword&nbsp; &quot;resources&quot;<\/em><\/li>\n<li><em>site:.edu keyword&nbsp; &quot;<\/em><em>links&quot;<\/em><\/li>\n<li><em>site:.co.uk keyword&nbsp; &quot;suggested sites&quot;<\/em><\/li>\n<li><em>site:.com.au keyword&nbsp; &quot;recommended sites&quot;<\/em><\/li>\n<\/ul>\n<\/p>\n<\/p><\/div>\n<\/section>\n<section id=\"google-your-first-search-term\">\n<h2> Google your first search term <\/h2>\n<div class=\"text-content\">\n<p>Broken link scraping&nbsp;is about covering a ton of ground, not about being careful. Get to it by shoving your first query into Google, opening the first page up and copying the URLs into a text document.<\/p>\n<\/p><\/div>\n<div class=\"image-content\">\n<figure>\n   <a href=\"https:\/\/www.process.st\/templates\/wp-content\/uploads\/2024\/02\/jw8W__PL_rVHmMMnYS9JWQ.png\" alt=\"Google your first search term\" target=\"_blank\" rel=\"noopener\"> <img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/www.process.st\/templates\/wp-content\/uploads\/2024\/02\/jw8W__PL_rVHmMMnYS9JWQ.png\" \/> <\/a><br \/>\n  <\/figure>\n<\/p><\/div>\n<div class=\"text-content\">\n<p>For a quicker method, put your query in <a href=\"http:\/\/ninjaseotools.com\/google-url-scraper.php\" target=\"_blank\" rel=\"nofollow noopener\">Ninja SEO Tools' Google URL Scraper<\/a> and get all of the results as a text file for less time spent copy-pasting.<\/p>\n<\/p><\/div>\n<div class=\"image-content\">\n<figure>\n   <a href=\"https:\/\/www.process.st\/templates\/wp-content\/uploads\/2024\/02\/oI-s4eG8ZOLyQGHCjNRAqA.png\" alt=\"Google your first search term\" target=\"_blank\" rel=\"noopener\"> <img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/www.process.st\/templates\/wp-content\/uploads\/2024\/02\/oI-s4eG8ZOLyQGHCjNRAqA.png\" \/> <\/a><br \/>\n  <\/figure>\n<\/p><\/div>\n<\/section>\n<section id=\"search-each-resulting-url-in-seo-spider\">\n<h2> Search each resulting URL in SEO Spider <\/h2>\n<div class=\"text-content\">\n<p>Now you should have several Google results (99 in one go if you used the Ninja SEO Tool I mentioned).<\/p>\n<p>Open up your list next to SEO Spider, and start pasting in.<\/p>\n<\/p><\/div>\n<div class=\"image-content\">\n<figure>\n   <a href=\"https:\/\/www.process.st\/templates\/wp-content\/uploads\/2024\/02\/jTdPz8ul9mjmnXLmPc5Pzg.png\" alt=\"Search each resulting URL in SEO Spider\" target=\"_blank\" rel=\"noopener\"> <img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/www.process.st\/templates\/wp-content\/uploads\/2024\/02\/jTdPz8ul9mjmnXLmPc5Pzg.png\" \/> <\/a><br \/>\n  <\/figure>\n<\/p><\/div>\n<div class=\"text-content\">\n<p>Make sure you have SEO Spider configured in 'Spider' mode, and select the '4xx' filter from the righthand menu:<\/p>\n<\/p><\/div>\n<div class=\"image-content\">\n<figure>\n   <a href=\"https:\/\/www.process.st\/templates\/wp-content\/uploads\/2024\/02\/gDTIEQPA3qdgXDfEGyBMZA.png\" alt=\"Search each resulting URL in SEO Spider\" target=\"_blank\" rel=\"noopener\"> <img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/www.process.st\/templates\/wp-content\/uploads\/2024\/02\/gDTIEQPA3qdgXDfEGyBMZA.png\" \/> <\/a><br \/>\n  <\/figure>\n<\/p><\/div>\n<\/section>\n<section id=\"note-all-broken-links-in-your-sheet\">\n<h2> Note all broken links in your sheet <\/h2>\n<div class=\"text-content\">\n<p>Fill in links in the 'unprocessed' tab for every broken link.<\/p>\n<\/p><\/div>\n<\/section>\n<section id=\"repeat-until-you-have-your-target\">\n<h2> Repeat until you have your target <\/h2>\n<div class=\"text-content\">\n<p>Around 100 <em>seed<\/em> links should do for a campaign.<\/p>\n<\/p><\/div>\n<\/section>\n<section id=\"finding-sites-hosting-these-broken-links-with-ahrefs\">\n<h2> Finding sites hosting these broken links with Ahrefs: <\/h2>\n<\/section>\n<section id=\"search-the-link-in-ahrefs-site-explorer\">\n<h2> Search the link in Ahrefs Site Explorer <\/h2>\n<div class=\"text-content\">\n<p>Ahrefs Site Explorer will show you all sites which link to your newly discovered broken URL.<\/p>\n<p>Go to Site Explorer and search your link, making sure you've got 'URL' selected on the dropdown.<\/p>\n<\/p><\/div>\n<div class=\"image-content\">\n<figure>\n   <a href=\"https:\/\/www.process.st\/templates\/wp-content\/uploads\/2024\/02\/jCV6lK1ABs9R7vHuKCZKcw.png\" alt=\"Search the link in Ahrefs Site Explorer\" target=\"_blank\" rel=\"noopener\"> <img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/www.process.st\/templates\/wp-content\/uploads\/2024\/02\/jCV6lK1ABs9R7vHuKCZKcw.png\" \/> <\/a><br \/>\n  <\/figure>\n<\/p><\/div>\n<\/section>\n<section id=\"check-the-backlinks-pointing-to-the-broken-url\">\n<h2> Check the backlinks pointing to the broken URL <\/h2>\n<div class=\"text-content\">\n<p>Hopefully, for every URL you find you'll find several unfortunate sites linking to it. These are the main targets.<\/p>\n<p>Click 'Backlinks' inside Ahrefs to filter down results to show sites that are pointing to your broken link:<\/p>\n<\/p><\/div>\n<div class=\"image-content\">\n<figure>\n   <a href=\"https:\/\/www.process.st\/templates\/wp-content\/uploads\/2024\/02\/tBxyQHoIXGzki1lrtsdIAA.png\" alt=\"Check the backlinks pointing to the broken URL\" target=\"_blank\" rel=\"noopener\"> <img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/www.process.st\/templates\/wp-content\/uploads\/2024\/02\/tBxyQHoIXGzki1lrtsdIAA.png\" \/> <\/a><br \/>\n  <\/figure>\n<\/p><\/div>\n<\/section>\n<section id=\"add-these-backlinks-to-also-linked-from-columns\">\n<h2> Add these backlinks to 'Also linked from' columns <\/h2>\n<div class=\"text-content\">\n<p>In Sheets, you'll see the 'Also linked from' columns. For however many backlinks there are listed there, list them in their own column on the row that's for the seed broken link, like this:<\/p>\n<\/p><\/div>\n<div class=\"image-content\">\n<figure>\n   <a href=\"https:\/\/www.process.st\/templates\/wp-content\/uploads\/2024\/02\/j2mW3MJznqiuzkKG1T9Hyw.png\" alt=\"Add these backlinks to 'Also linked from' columns\" target=\"_blank\" rel=\"noopener\"> <img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/www.process.st\/templates\/wp-content\/uploads\/2024\/02\/j2mW3MJznqiuzkKG1T9Hyw.png\" \/> <\/a><br \/>\n  <\/figure>\n<\/p><\/div>\n<\/section>\n<section id=\"repeat-for-every-seed-backlink-you-found-in-the-first-section\">\n<h2> Repeat for every seed backlink you found in the first section <\/h2>\n<div class=\"text-content\">\n<p>Continue building out the 'Unprocessed' section of your sheet by finding sites that link TO the broken URLs we originally found.<\/p>\n<\/p><\/div>\n<\/section>\n<section id=\"finding-the-site-owners-contact-information\">\n<h2> Finding the site owner's contact information: <\/h2>\n<\/section>\n<section id=\"fill-in-the-site-owners-email-addresses\">\n<h2> Fill in the site owner's email addresses <\/h2>\n<div class=\"text-content\">\n<p>Now you'll be left with a sheet with hundreds of broken links, it's time to move them over to the 'Processed' sheet and find the contact details for every site owner with a broken link you want to replace.<\/p>\n<p>For every broken link, make a new column in the Processed tab and build it out until it's done. Then, <strong>use the process I'm going to link below<\/strong> to find their email addresses:<\/p>\n<p><strong><a href=\"https:\/\/www.process.st\/templates\/how-to-find-almost-anyones-email-address\/\" rel=\"nofollow noopener\" target=\"_blank\">How to Find Almost Anyone's Email Address<\/a><\/strong><\/p>\n<p>After you've filled in the emails, go ahead and progress to the next template:<\/p>\n<p><strong><a href=\"https:\/\/www.process.st\/templates\/outreach-for-broken-link-building\/\" target=\"_blank\" rel=\"nofollow noopener\">Broken Link Building: Outreach<\/a><\/strong><\/p>\n<\/p><\/div>\n<\/section>\n","protected":false},"excerpt":{"rendered":"<p>Initial setup: You're going to need: Screaming Frog SEO Spider (free for searches of 500 links at a time) Google account Ahrefs account (to use Site Explorer) Finding seed broken links with Google: Create a new Google Sheet template Use this template:&nbsp;https:\/\/docs.google.com\/spreadsheets\/d\/1jQirKt3N1seiV-2KWHcTXI2iDywyg0U_IxggtQWe1E0\/pub?output=csv Make a copy in your own account. You'll notice it has 2 tabs, [&hellip;]<\/p>\n","protected":false},"author":3,"featured_media":5856,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"ep_exclude_from_search":false,"cover_icon_emoji":"","cover_icon_url":"","tasks_count":"15","template_description":"A step-by-step guide to quickly scraping broken links which you can contact the site owner about and get your link listed instead.","template_id":"oWUHMSR0DjHOuuU__AFBrA","task_0":"Initial setup:","task_slug_0":"initial-setup","task_1":"Finding seed broken links with Google:","task_slug_1":"finding-seed-broken-links-with-google","task_2":"Create a new Google Sheet template","task_slug_2":"create-a-new-google-sheet-template","task_3":"Create a list of search terms","task_slug_3":"create-a-list-of-search-terms","task_4":"Google your first search term","task_slug_4":"google-your-first-search-term","task_5":"Search each resulting URL in SEO Spider","task_slug_5":"search-each-resulting-url-in-seo-spider","task_6":"Note all broken links in your sheet","task_slug_6":"note-all-broken-links-in-your-sheet","task_7":"Repeat until you have your target","task_slug_7":"repeat-until-you-have-your-target","task_8":"Finding sites hosting these broken links with Ahrefs:","task_slug_8":"finding-sites-hosting-these-broken-links-with-ahrefs","task_9":"Search the link in Ahrefs Site Explorer","task_slug_9":"search-the-link-in-ahrefs-site-explorer","task_10":"Check the backlinks pointing to the broken URL","task_slug_10":"check-the-backlinks-pointing-to-the-broken-url","task_11":"Add these backlinks to 'Also linked from' columns","task_slug_11":"add-these-backlinks-to-also-linked-from-columns","task_12":"Repeat for every seed backlink you found in the first section","task_slug_12":"repeat-for-every-seed-backlink-you-found-in-the-first-section","task_13":"Finding the site owner's contact information:","task_slug_13":"finding-the-site-owners-contact-information","task_14":"Fill in the site owner's email addresses","task_slug_14":"fill-in-the-site-owners-email-addresses","task_15":"","task_slug_15":"","task_16":"","task_slug_16":"","task_17":"","task_slug_17":"","task_18":"","task_slug_18":"","task_19":"","task_slug_19":"","task_20":"","task_slug_20":"","task_21":"","task_slug_21":"","task_22":"","task_slug_22":"","task_23":"","task_slug_23":"","task_24":"","task_slug_24":"","task_25":"","task_slug_25":"","task_26":"","task_slug_26":"","task_27":"","task_slug_27":"","task_28":"","task_slug_28":"","task_29":"","task_slug_29":"","task_30":"","task_slug_30":"","task_31":"","task_slug_31":"","task_32":"","task_slug_32":"","task_33":"","task_slug_33":"","task_34":"","task_slug_34":"","task_35":"","task_slug_35":"","task_36":"","task_slug_36":"","task_37":"","task_slug_37":"","task_38":"","task_slug_38":"","task_39":"","task_slug_39":"","task_40":"","task_slug_40":"","task_41":"","task_slug_41":"","task_42":"","task_slug_42":"","task_43":"","task_slug_43":"","task_44":"","task_slug_44":"","task_45":"","task_slug_45":"","task_46":"","task_slug_46":"","task_47":"","task_slug_47":"","task_48":"","task_slug_48":"","task_49":"","task_slug_49":"","task_50":"","task_slug_50":"","task_51":"","task_slug_51":"","task_52":"","task_slug_52":"","task_53":"","task_slug_53":"","task_54":"","task_slug_54":"","task_55":"","task_slug_55":"","task_56":"","task_slug_56":"","task_57":"","task_slug_57":"","task_58":"","task_slug_58":"","task_59":"","task_slug_59":"","task_60":"","task_slug_60":"","task_61":"","task_slug_61":"","task_62":"","task_slug_62":"","task_63":"","task_slug_63":"","task_64":"","task_slug_64":"","task_65":"","task_slug_65":"","task_66":"","task_slug_66":"","task_67":"","task_slug_67":"","task_68":"","task_slug_68":"","task_69":"","task_slug_69":"","task_70":"","task_slug_70":"","task_71":"","task_slug_71":"","task_72":"","task_slug_72":"","task_73":"","task_slug_73":"","task_74":"","task_slug_74":"","task_75":"","task_slug_75":"","task_76":"","task_slug_76":"","task_77":"","task_slug_77":"","task_78":"","task_slug_78":"","task_79":"","task_slug_79":"","task_80":"","task_slug_80":"","task_81":"","task_slug_81":"","task_82":"","task_slug_82":"","task_83":"","task_slug_83":"","task_84":"","task_slug_84":"","task_85":"","task_slug_85":"","task_86":"","task_slug_86":"","task_87":"","task_slug_87":"","task_88":"","task_slug_88":"","task_89":"","task_slug_89":"","task_90":"","task_slug_90":"","task_91":"","task_slug_91":"","task_92":"","task_slug_92":"","task_93":"","task_slug_93":"","task_94":"","task_slug_94":"","task_95":"","task_slug_95":"","task_96":"","task_slug_96":"","task_97":"","task_slug_97":"","task_98":"","task_slug_98":"","task_99":"","task_slug_99":"","footnotes":""},"categories":[7],"tags":[],"class_list":["post-5855","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-miscellaneous"],"acf":[],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.process.st\/templates\/wp-json\/wp\/v2\/posts\/5855","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.process.st\/templates\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.process.st\/templates\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.process.st\/templates\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/www.process.st\/templates\/wp-json\/wp\/v2\/comments?post=5855"}],"version-history":[{"count":0,"href":"https:\/\/www.process.st\/templates\/wp-json\/wp\/v2\/posts\/5855\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.process.st\/templates\/wp-json\/wp\/v2\/media\/5856"}],"wp:attachment":[{"href":"https:\/\/www.process.st\/templates\/wp-json\/wp\/v2\/media?parent=5855"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.process.st\/templates\/wp-json\/wp\/v2\/categories?post=5855"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.process.st\/templates\/wp-json\/wp\/v2\/tags?post=5855"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}