Research off the shelf solutions for a back end batch system (scraping application)

  • Status: Closed
  • Prize: $170
  • Entries Received: 2
  • Winner: TheGuyver2040

Contest Brief

The task is to research and suggest possible off the shelf solutions for a back end batch system (scraping application) that fulfill the following requirements:

Business Objectives:
Get competitors product information for branded shoes on a daily basis. Information like Product name, Product colorname, other product attributes example materials, list of competitors, Competitor prices (Black and discounted prices)).

Proposed Work Flow:
The system fetches a product list from our system with EAN codes (for identification). The system then uses the EAN in google shopping to fetch product information and prices per competitor.

Required Features:
Scraping engine to handle over 15.000 requests per day. The normal request per day will be 50-500. The information should be structured on model level and SKU / EAN level (ie we send over/fetch one size/EAN to scrape).


Inputs:
There will be a product feed into the system from our product system. Alternativly we want to be able to fetch the data through a feed / API.


Dependencies:
The system should have an easy way to export / import information preferably using some API / feed.

Integrations:
There should be an API available. To fetch the scraped products. And possibly what products of interest to scrape.

Server / Hosting:
AWS on a Kubernetes server, if hosted on our servers. Otherwise software as a service is fine.

User Count:
Performance is hard to estimate but roughly 50-500 daily requests but up to 15.000 when going through our total range of products.

Deliverable:
A summary of the system you have found with a short explanation of its functionalities and how it can be used to fulfil our requirements

Recommended Skills

Employer Feedback

“It was great! ”

Profile image Footway, Sweden.

Top entries from this contest

View More Entries

Public Clarification Board

No messages yet.

How to get started with contests

  • Post your contest

    Post Your Contest Quick and easy

  • Get tons of entries

    Get Tons of Entries From around the world

  • Award the best entry

    Award the best entry Download the files - Easy!

Post a Contest Now or Join us Today!