Find Jobs
Hire Freelancers

Scrapy Scraping and Web Crawling parallel execution

$250-750 USD

Completed
Posted over 1 year ago

$250-750 USD

Paid on delivery
Hey, I need a web crawler for a certain website that discovers links by itself and manages to intelligent scrape some content from it. Requirements: - Can run multiple instances at the same time (so 2/3 crawler for the same website at the same time) - Prevents the crawling of a link twice within a certain period (crawl a link only every 2 weeks) - Crawl (find new links) - Extract around 10 data points - Save to database - Execute JavaScript - Use Proxies (provided by me) - Has restriction of crawls per minute - Uses state-of-the-art methods for hiding that it is not a person (browser fingerprint etc.) Please state your experience and also how you think you can solve the need to manage the multiple instances and prevent crawling twice. Thnaks.
Project ID: 34951087

About the project

58 proposals
Remote project
Active 2 yrs ago

Looking to make some money?

Benefits of bidding on Freelancer

Set your budget and timeframe
Get paid for your work
Outline your proposal
It's free to sign up and bid on jobs
Awarded to:
User Avatar
$0 USD in 15 days
5.0 (24 reviews)
5.3
5.3
58 freelancers are bidding on average $513 USD for this job
User Avatar
Hi sir, This is Lin and I am scraping expert, I scraped 600+ websites, please check my reviews then you will know. I will create scraping tool with PHP & CURL that extracts all requested data, And provides data in the required format (JSON, CSV, EXCEL, SQL...). Can we discuss more details about this project? Then I can provide sample data/script, we can start project after you checked sample. Thanks, Lin
$600 USD in 7 days
5.0 (428 reviews)
8.2
8.2
User Avatar
Hi. We already have a tool that does all that you mentioned, all we need todo is implement specific website you have and run it. What is the website ? Thanks,.
$750 USD in 7 days
5.0 (142 reviews)
7.9
7.9
User Avatar
Hello, I have 12+ years of experience in Python and have worked on a lot of Django, Flask, AI/ML, ... projects. I can share some demos if you are interested. Also, I work as a fulltime freelancer and have enough time to assign to your project. Thanks, Helmot
$500 USD in 7 days
4.8 (151 reviews)
7.8
7.8
User Avatar
Hi Good afternoon , How are you? I just saw your job posting . I see you have been looking for someone experience with these technologies Scrapy, Web Scraping, Data Scraping, Web Crawling and Python. I believe this is some thing I can help you with because I have 9 years of experience working with these tech stacks and 8 years of experience working on freelancer.com which you can confirm from my profile: https://www.freelancer.com/u/AwaisChaudhry I have few questions: 1. These are all the requirements of your job or do you have more? If yes, Please provide detailed requirements. 2. Do you currently have anything done like Wireframes, UI/UX? Or this job has to be done from scratch? 3. What is the timeline to get this job done? Why Choose Me? 1. I have done more than 250 major projects only on freelancer.com. 2. I have not received a single bad feedback since last 5-6 years. 3. You will find 5 star feedback on last 100+ major projects which shows my clients are happy with my work. Portfolio: https://www.freelancer.com/u/AwaisChaudhry Timings: 9am - 9pm Eastern Time (I work as a full time freelancer) Lets start the chat and discuss your job in detail and we will continue from there. Regards, Awais
$750 USD in 22 days
5.0 (9 reviews)
6.7
6.7
User Avatar
Hello, there. I am very interested in your proposal. I can instantly help you with your starting project with a successful completion. As a professional software developer, I ensure for a perfect ongoing project. You will never be disappointed in me for sure. Looking forward to meeting you on chat. Thank you. Darrel.
$500 USD in 7 days
5.0 (18 reviews)
6.4
6.4
User Avatar
Hi there,I'm biddin on your project "Scrapy Scraping and Web Crawling parallel execution" I have read your project description and i'm an expert in Python and machine learning therefore i can do this project for you perfectly.I still have a few questions. please leave a message on my chat so we can discuss the budget and deadline of the project. Thanks. .. .
$750 USD in 3 days
4.9 (25 reviews)
6.5
6.5
User Avatar
Hello, this is Koki from Japan who has been working with Python and Web Scraping development for over 8 years now. I have checked your project description carefully and I think that I can help you to complete this project fully 100% sure to satisfy your requirement. I'd like to have a brief chat or call to discuss further details about your project soon. Looking forward to working with you together on this project. Thanks. Koki.
$1,000 USD in 5 days
4.7 (15 reviews)
6.9
6.9
User Avatar
Hello, this is Luka from Serbia who has been working with pyhon development for over 7 years now. I have checked your project description carefully and I think that I can help you to complete this project Looking forward to working with you together on this project. Thanks. Luka
$749.99 USD in 5 days
5.0 (6 reviews)
5.5
5.5
User Avatar
Hi, I am senior web scraping deveoper. Python webScraping is my top skill. I have already developed many( more than 100) web scraping tools using python. So I will provide you awesome result . Please contact me for our project. Really want to work for you.
$500 USD in 7 days
5.0 (16 reviews)
5.6
5.6
User Avatar
Hello @findoo. I'm interested in your project because I'm experienced web scraper I have been working on web scraping using python for 7 years Especially I'm familiar with scrapy, selenium, BeutifulSoup, request python package. My strong skill is fast scraping. I hope we discuss your project briefly. I can start working right now. Best & Regards Serhii
$577 USD in 7 days
5.0 (6 reviews)
5.7
5.7
User Avatar
✓ Hi, How are you ? I have more than 5 years experience in ✔️Python, web scraping, selenium, expert CSV, pandas..✔️. I can start the work right now. Looking forward to hear more about the project via chat. Best regards!
$500 USD in 7 days
5.0 (9 reviews)
5.2
5.2
User Avatar
Hi I have 4 years of expirence in creating scraping and webcrwaling programs and can do this task for you I understand your requirements completely and I know methods to run multiple instance at the same time without crawling same link twice and have good experience in avoiding bot detection program I will make sure the quality work under deciesed timeline and budget Thanks.
$500 USD in 20 days
5.0 (4 reviews)
5.4
5.4
User Avatar
Dear client I have read your description carefully and I am confident about your work. As a web scraping expert, I have over 7 years of experience in web scraping. I have collected information from social sites, e-commerce sites and booking sites and analyze information. Also I have scraped the betting site and predicted the result. With my extensive experience, I can get solid knowledge in web scraping, automation and bots. I used python(selenium, beautifulsoup, requests), and node.js(axios, cheerio, puppeteer) for scraping. I am sure I can fulfill your requirements. I am confident that you will find my hands-on experience. Thank you for your time and consideration. Please let me know if there is anything further you need from me. Thanks.
$500 USD in 7 days
4.8 (13 reviews)
5.2
5.2
User Avatar
Hi there, ★★★ Scrapping / Python / Selenium Expert ★★★ 6+ Years of Experience ★★★ I've read requirements and ready to write script to scrape data from website. Some major works we do: ✔️ Product Websites Scraping: eCommerce (Shopify, eBay, Amazon, AliExpress. etc) ✔️ Advertising or Ads posting sites (gumtree, Rightmove, Asian metal. etc) ✔️ Login required websites ✔️ Process Automated Tasks (Automatic download file from any website pdf, CSV, etc) ✔️ Software and Scripts Development ✔️ With Browser load and Without Browser load ✔️ Web research and Data Entry work as well, ✔️ Scrapped Datastore into multiple form SQL server, CSV, Excel any many more like JSON. We use different tools and techniques to scrape the websites depending on the kind of task but mainly Python, C# Desktop Application and Selenium and Html Agility. Note: Bid is tentative. To provide you the best quote, i'll be glad to discuss requirements so let's have a chat. Best Regards, TechPlus Team
$750 USD in 7 days
4.3 (21 reviews)
5.8
5.8
User Avatar
Hello client, First, receive warm greetings, and hoping you are good, I welcome you to the home of Amazing services and quality. I am an expert in named skills,over 9 years of experience, After keenly reading your description and being in possession of all clearly stated required skills as this is my area of professional specialization having the above qualifications, I hereby humbly request you to consider my bid for quality and professional work. It is my duty to ensure 100% satisfaction to you as my client. No timeline and quality excuses. Unlimited revisions until all client requirements are met as required. affordable, professional & quality services are guaranteed always. Message me we discuss more the project. thank you.
$750 USD in 10 days
4.8 (3 reviews)
5.4
5.4
User Avatar
Hi I'm Python/Web Scraping/Automation Expert. I’ve gone through your requirements, it perfectly matches my skills. I am interested to work with you on this project. I've been working on scraping e-commerce websites, social media posts from(Facebook, twitter etc..), Google searches.. I can do it using Python(selenium/requests/bs4 libraries depending on the website), and the data can be extracted to csv as you provided. Ready to assist you and start working right away. Best Regards
$500 USD in 7 days
5.0 (13 reviews)
4.9
4.9
User Avatar
Dear Sir I just read your description more detail and it sounds like you want scraper. I am very motivated in this project. I have much experiences of scraping software development and have been working for 7 years. I already have developed many odds scraper for sports betting bookies. I am particular proud of my skills. This job is an ideal match for my skills and experiences. I will pull my weight to this project. I would like to discuss about your project more specifics after contacting. Best Regard.
$500 USD in 7 days
5.0 (4 reviews)
4.4
4.4
User Avatar
Hi, I am Web scrapping expert. I will use beautiful Soup and selenium for scrapping website. I have scrapped 400+website. Please, check my skills. I hope to discuss about it with you. Thanks.
$250 USD in 2 days
4.8 (11 reviews)
4.1
4.1
User Avatar
Hi, How are you? Upon reading brief details I believe it's a doable job I have great experience doing projects with Web Crawling, Scrapy, Python, Web Scraping and Data Scraping. I am confident about getting this done. While I believe we need to have a detailed discussion about it. I work as per Eastern time daily so that would be good for us to catch up on this job, Let's discuss this further and get started for you as soon as possible. Thanks! Muhammad
$250 USD in 1 day
5.0 (4 reviews)
4.0
4.0
User Avatar
Hello: It is a pleasure to communicate with you. My name is Anthony Muñoz, I am the lead engineer for DSPro IT agency and I would like to offer you my professional services. I have more than 10 years of working as a Backend and Software developer, I have successfully completed numerous jobs similar to yours on this and other Freelance platforms therefore, and after carefully reading the requirements of your project, I consider this job to be suitable to my area of knowledge and skills. I would love to work together to make this project a reality. I greatly appreciate the time provided and I remain pending for any questions or comments. Feel free to contact me. Greetings
$508 USD in 7 days
5.0 (1 review)
3.8
3.8

About the client

Flag of GERMANY
Essen, Germany
5.0
24
Payment method verified
Member since Jan 12, 2010

Client Verification

Thanks! We’ve emailed you a link to claim your free credit.
Something went wrong while sending your email. Please try again.
Registered Users Total Jobs Posted
Freelancer ® is a registered Trademark of Freelancer Technology Pty Limited (ACN 142 189 759)
Copyright © 2024 Freelancer Technology Pty Limited (ACN 142 189 759)
Loading preview
Permission granted for Geolocation.
Your login session has expired and you have been logged out. Please log in again.