Software Engineer with Python, Web Scraping, and RDBMS Expertise Needed

Closed Posted 1 year ago Paid on delivery
Closed

We are looking for a Python software engineer with lots of web scraping experience to help us scrape and create a large-scale database. We will use a variety of APIs, scraping tools, and need complex parsing to crawl the website pages, identify the correct pages, and then scrap the data. We will be collecting both regular numeric, text, and other data types and also want to find and download PDFs, images, and more (unstructured data).

We will need to visit and crawl 70,000+ web sites and find pages with data by category.

This project will create multiple SQL databases: one database with structured information and another with free-form unstructured information. The database with structured information will get some data through API calls and other data by scraping website pages.

Because this will evolve into a complex system we want someone with no less than five years experience programming and ideally 10+ years who understands enterprise systems, servers and architecture, not just scraping.

We are building an industrial strength, custom scrapping system that takes data from thousands of source websites and routes it into multiple SQL RDBMS. The ideal candidate will have experience with multiple scrapers like Scrapy, Selenium, Beautiful Soup, and Requests.

Since we have a list of thousands of different websites to scrape, and each of these websites have a different structure, we need to build a custom-made way to search, find, and select the correct pages and then scrape the information we need (i.e. resumes, tables, graphics).

We want to build a way to scrape pages automatically by looking for keywords on the hundreds of thousands of pages we will scrape. There may be a list of 50 categories of information we will search for in time at these URLs. And it may make sense to use or call different scraping tools and build custom ones. And this will be set up to also compare and detect changes in the data fields (versus the last visit/scrape) and flag these changes.

The process scraping unstructured information will find and scrape web pages to extract a wide variety of information like project roadmaps, organization mission statements, investor information, team biographies/resumes, financial data, etc.

This could evolve into an ongoing project either part-time or full-time after we build the prototype/MVP product.

Python Web Scraping Data Mining SQL Data Scraping

Project ID: #33430956

About the project

35 proposals Remote project Active 1 year ago

35 freelancers are bidding on average $36/hour for this job

mrogowski

Hi, there! I've been working as a Python Software Engineer for the last 7 years and I have 10+ years of experience on both software and hardware engineering. I'm pretty sure I'm the right fit for you. Let's schedule More

$60 USD / hour
(12 Reviews)
7.8
MashoodurRehman1

Python developer - Web Scrapping - RDBMS I have read your job description and I am pretty sure that I can complete every bit of your requirements. Further details and cost can be discus

$25 USD / hour
(141 Reviews)
7.2
datascientist90

I'm a senior Python & ML developer and owner & founder of Dedeoglu Dev Company. Kindly send me a message to get in touch with me, Thanks, Yusuf.

$38 USD / hour
(26 Reviews)
6.6
Koki1216

Hello, this is Koki from Japan who has been working with Python, Web Scraping development for over 7 years now. I have checked your project description carefully and I think that I can help you to complete this project More

$50 USD / hour
(15 Reviews)
6.9
sapnathakur14

Hello, How are you? I read your job post, and I am interested to work with you on this project, as I have relevant skills set, you can check my profile for your surety. Rest we will finalize everything after complete More

$25 USD / hour
(11 Reviews)
6.1
freelancerIrvan

Hi there. I know where this project was posted initially, because I also was applied to this job there also anyhow, I have the experience in building some crawling system. so I can handle this project. If you got this More

$30 USD / hour
(15 Reviews)
5.6
techplusintl

Hi there, ★★★ Scrapping / Python / Selenium Expert ★★★ 10+ Years of Experience ★★★ I've read requirements and ready to work on your project. Some major works we do: ✔️ Product Websites Scraping: eCommerce (Shopify, eB More

$40 USD / hour
(28 Reviews)
5.9
romanvaraksin

Hello! Thank you for your time on checking my proposal. From reading your post and visiting your website, I can see that you are finding a web scraper who has a currency view. From my deep experience in web scraping More

$25 USD / hour
(13 Reviews)
5.2
abdulsamad724

Dear Customer. I am an expert in Python, Excel, Web Scraping, Data Mining and I assure you that you will be 100% satisfied with the results of my work. I will deliver you a full scope of the services you would like to More

$25 USD / hour
(5 Reviews)
4.2
varonedgar

⭐ Expert of Python and WebScrapping HERE ⭐ Hi Client! I am Varon, rich experienced software expert from Colombia. I noticed that I am appropriate to this project. As a skillful software developer, I have rich experie More

$25 USD / hour
(6 Reviews)
4.4
hnutweblera

Hello there. I have good experience with python scrapping and automation using Scrapy, Selenium, Beautiful Soup. So can help your project. Hope more discuss with you. Regards.

$35 USD / hour
(5 Reviews)
5.0
VladProkopchuk

Hello I am senior web scrapping expert I have worked this filed for over 10 years Yes, We have to know python, selenium, beautifulsoup and many other fields of knowledge I am very familiar with these fields, so I am th More

$38 USD / hour
(7 Reviews)
4.3
manojguragain184

Dear, Client, How are you doing well? I am ready for your long term scraping project. As a python scraping and bot developer, I can scrape any site without blocking. -My service:    1: Scraping API or Html or JavaScri More

$30 USD / hour
(8 Reviews)
4.5
AbhishekSingh08

I have gone through your requirement to scrape lots of websites. I am an EXPERT in building scraping tools /scripts. Hence, I can SURELY work on your project. I am having 7 YEARS of EXPERIENCE in developing PHP-PYTHON More

$40 USD / hour
(2 Reviews)
3.8
melmougith

Hi, I think that you need a developer with good python skills for scrapping, but also an analytical mind that understand perfectly how to make the best out of the scrapped information. This would be by good databases More

$30 USD / hour
(10 Reviews)
3.9
alexdewbest

Warm Greetings, I go through job description and read it carefully, I got exactly what you are looking for. I have rich experience and good skills in required skills for your project such as Python, Scrapy, Selenium, More

$38 USD / hour
(1 Review)
3.4
vivekbharti900

looking your requirement I feel I can do it and finish your job in proper way. I have made one traveling website and its running with proper condition.

$25 USD / hour
(0 Reviews)
0.0
OthmenBraham

Hi, i'm an expert in highly responsive website with optimale web technologies, please check my feedback then you will know. i will work this project with heigh scaping technolgie online, with buttons, progress bar, mul More

$25 USD / hour
(0 Reviews)
0.0
doddyharic

Hi there, It may be strange if look into my profile and find out that I haven't had any sales yet. If that's the case, you can ignore my proposal. I am quite interested in the job since I am already done quite a lot More

$25 USD / hour
(0 Reviews)
0.0
abdf2010

Hi, It is easy, I can do on time. I work online, where you can track progress of your project. I have 6 years of experience in development(websites, web applications, mobile apps, desktop applications, I/UX), using PHP More

$25 USD / hour
(0 Reviews)
0.0