Scrapy Jobs

I need a clean, one-time extraction of every registered agent listed on the Kerala RERA portal. The scope is limited to the publicly displayed agent name plus all available contact details—phone numbers, email addresses, and office addresses. No licence-status fields or property listings are required. Any stack is fine—Python (BeautifulSoup, Scrapy, Selenium), Node.js, or a headless browser workflow—as long as it handles pagination, hidden rows, or JavaScript-rendered tables and respects polite scraping practices. Deliverables • An Excel workbook (.xlsx) containing one row per agent and clearly labelled columns for Name, Phone, Email, and Address • Data fully deduplicated, UTF-8 compliant, and free of blank placeholders • A short note on the appro...

I need two separate, non-blocking crawlers—one targeting Amazon, the other Flipkart—each operating with our seller accounts. Every 24 hours the system must fetch fresh data for the ASINs I supply: product name, seller name, star rating, ratings count, MRP, and current selling price. The information should be written straight into a MySQL database and the entire cycle has to repeat indefinitely without manual intervention. Reliability is critical. If either crawler encounters captchas, rate limits, network drops, or any other interruption, it should automatically recover (self-healing logic) and continue from the exact point of failure. Should a fault persist beyond the retry logic, I want an immediate email alert with a concise error report so I can act quickly. Please struct...

I need a robust web-scraping solution that automatically collects product information from several e-commerce websites. The focus is on two key data points: • Product name and full description • Customer reviews and ratings Price and availability are not required this time, so the crawler can ignore any endpoints related to stock or cost. Please build the script so I can run it on demand and easily point it at new store URLs in the future. Python with BeautifulSoup, Scrapy, or a similar framework suits me fine, as long as the code is clean, well-commented, and leverages polite scraping practices (respectful delays, user-agent rotation, handling captchas when possible). Deliverables: 1. Working scraper code with clear setup instructions 2. Sample CSV or JSON export c...

Project Title: Large-Scale Course Data Scraping from Udemy, Coursera, and YouTube Project Overview: I am looking for an experienced data scraping specialist who can collect a large dataset of online courses and tutorials from platforms such as Udemy, Coursera, and YouTube. The dataset should include both technical and non-technical tutorials. The objective is to build a structured dataset that can be used for research and analytics purposes. Project Requirements: The freelancer will scrape approximately 200,000 course/tutorial records from the following platforms: - Udemy - Coursera - YouTube The collected dataset should contain the following information for each record: 1. Course / Tutorial Title 2. Course URL / Video URL 3. Platform Name (Udemy / Coursera / YouTube) 4...

We need a robust yet lightweight script that can automatically pull business details from a publicly accessible government website. The information to capture will centre on business details, such as registration numbers, business name, registration date, address, etc. The workflow should: • Navigate every relevant section of the site (pagination, search filters, subsidiary pages). • Extract the required fields accurately • Export clean, structured data to CSV and JSON A Python solution leveraging requests/BeautifulSoup or Scrapy is preferred, but I’m open to other dependable stacks if they handle rate-limits, retries, and potential CAPTCHA gracefully. The script must be easy to rerun on demand, with clear instructions for environment setup and any dependencies...

Seeking a skilled web designer to create a clean, minimalistic website to showcase services. The website's primary function will be to highlight our offerings in a sleek, digestible format that engages visitors and inspires a sense of professionalism and quality. Features needed include: - A contact form to facilitate customer inquiries - Subscriber sign-up option for customers to receive updates and newsletters - A blog/news section to post updates and relevant content - An 'add listing' feature to display products effectively - Banner slide out et - Social Scroll It's crucial for the designer to have a keen eye for creating layouts that promote readability and user experience. Experience in minimalistic design can use template. Images & website examples will be...

# Python Developer (Scrapy / FastAPI / Docker) for Crawling Project We are looking for an experienced Python developer to build a structured web crawling system. This is not a large corporate project. It is a technically clean, well-defined system with a long-term perspective. We value clean architecture, stability, and understandable code. The system will process approximately 30,000 profiles per week and will be connected to an existing Laravel-based admin dashboard. --- ## Responsibilities * Develop a crawler using **Scrapy** * Build a small **FastAPI interface** (start / stop / status / statistics) * Integrate with **MariaDB** * Implement a media pipeline: download → local cache → upload to Wasabi (S3) * Implement hash-based change detection * Implement soft-delete logi...

I am looking for an experienced Web Scraping and Automation Developer to build an automated daily workflow for my business. I respond to over 100 government tenders a year and need to automate the discovery and document retrieval process. The Goal: Scrape 9 Australian Government tender websites daily for newly published tenders in two specific categories (UNSPSC 43000000 - IT, and 81000000 - Engineering/Research). Extract key details (Title, Agency, Closing Date, URL) and insert them into a centralized Google Sheet. Automatically download the associated Tender Documents and save them into uniquely named folders in my Google Drive. Periodically monitor these specific tenders for any newly published Addendums, and automatically download them to the respective Google Drive folder. The Website...

I have a list of roughly 1500 URLs—each coming from the same automotive website—that together cover the top 100 makes, models, grades and variants sold in Australia. I need every data point the site makes available for each of those vehicles, from the obvious specs such as year, make, model and variant right through to driveway prices, engine type, transmission, drive configuration, warranty details, fuel-economy figures, in-car technology features, seating layouts and any other attributes exposed on the page. The end goal is a clean, analysis-ready Excel workbook that lets me run market-wide comparisons, so consistency is critical: headings must be standardised, units normalised and categorical values written the same way across the entire sheet. I am happy for you to use P...

I need a reliable script that can pull live pricing details for car rentals from both the rental companies’ own sites and the big aggregator platforms in one pass. The goal is to feed it pickup / drop-off locations, dates, and driver age, then receive a clean CSV or JSON that lists vehicle class, daily and total price, currency, taxes & fees, and the URL it was scraped from. The scraper has to: • Navigate each target site automatically, including date pickers and location selectors. • Rotate user-agents / proxies or apply any other anti-bot tactics necessary to stay undetected. • Capture and log errors so a failed request never silently drops a row. • Be easy for me to rerun on demand—command-line or small web UI is fine, as long as setup is s...