Need to build a simple website crawler
Posted: 2/12/26 · Views: 944
- Position Type
- One-Time Gig (Project)
- Salary
- I need a Quote
- Skills/Tags
Hi There,
I'd like to build a web crawler. When someone types in a URL, the crawler should crawl the site and retrieve all the URLs of the site and export it to a CSV in addition to the URL, the following information is also required as columns in the export:
1 URL
2 Referring URL
3 Title
4 Meta Description
5 Status Code
6 Page Speed
7 Canonical URL
8 Redirect Location
9 Link Count
10 Page Authority
11 Issue Count
12 Word Count
13 Crawl Depth
14 No Follow
15 No Index
16 Robots
17 X-Robots
18 URL Length
19 Redirect To 404
20 Redirect Chain
21 Domain Authority
Please reply with a quote for how much you would want for this project.
Thanks
Suren