Need to build a simple website crawler

Posted: 2/12/26 · Views: 944

Position Type
One-Time Gig (Project)
Salary
I need a Quote
Skills/Tags
Description

Hi There,



I'd like to build a web crawler. When someone types in a URL, the crawler should crawl the site and retrieve all the URLs of the site and export it to a CSV in addition to the URL, the following information is also required as columns in the export:



1 URL

2 Referring URL

3 Title

4 Meta Description

5 Status Code

6 Page Speed

7 Canonical URL

8 Redirect Location

9 Link Count

10 Page Authority

11 Issue Count

12 Word Count

13 Crawl Depth

14 No Follow

15 No Index

16 Robots

17 X-Robots

18 URL Length

19 Redirect To 404

20 Redirect Chain

21 Domain Authority





Please reply with a quote for how much you would want for this project.



Thanks

Suren

Winnifred Satterfield

Want to apply?

Sign up or login here. If you don't already have an account, it's completely free.