Scrape righmove for commercial property data
Looking to scrape right move for commercial property data in addition to zoopla that you helped me with.
Rightmove commercial has a Get API to obtaining data in JSON format which iterated on index for a location and gathered , For. Eg
[login to view URL]
One of the technical challenge which will need solved is how to convert from city name(in [login to view URL]) to locationIdentifier required in the get URL. If I need to add areaNameRightmove similar to areaNameZoopla in var searchAreas([login to view URL]) I can do that, Let me know what to put in there.
The JS function will be passed an array of areas (var searchAreas).I suggest that GetJsonForZooplaCityList that you wrote be renamed & modified to return both property data from both zoopla & right move( is hoopla scrapping still working for you, they seem to have got wiser to [login to view URL]). This can be called with params to indicate what needs to be scrapped.
For e.g
.//passes array of sources to be scrapped , in the below e.g scrapes data for both
GetJsonForCityList(areasInViewPort, [“zoopla”,”rightmove”]).then(function(data) {
[login to view URL](data);//array of json representing properties scraped
});
//The above will give me flexibility to just scrape one if required.
=====
The end result object that you returned needs to be the same.
{"title":"Ropergate, Pontefract, WF8 1LY","image":"<a class=\"maps__results-info-box-image\" target=\"_blank\" href=\"[login to view URL]\"><img sizes=\"(min-width: 1200px) 24vw, (min-width: 758px) 32vw, (min-width: 512px) 49vw, 100vw\" srcset=\"[login to view URL] 150w, [login to view URL] 350w, [login to view URL] 550w, [login to view URL] 750w, [login to view URL] 1100w, [login to view URL] 1500w, [login to view URL] 1900w\" /></a>","types":["Leisure Property","Land","Land","Licensed & Leisure","Commercial Land"],"size":"14945 Sq Ft","sale_price":"£325,000.00","is_for_sale":true,"is_for_rent":false,"rent_price":"ROA","link":"[login to view URL]","lat_lon":"53.6901,-1.31336","must_see":false,"isProperty":true,"firstDetectedOnTimestamp":1533928244027,"firstDetectedOn":"10-Aug-2018 20:10:44","lastSeenOn":"18-Aug-2018 11:49:45","deleted":"false","source":"propertylink","lat":"53.6901","lng":"-1.31336","area":"Leeds","sizeInSqFt":"14945","lastSeenTimestamp":1534589385721}
====
The mapping for the above fields are as follows: (above property that needs to be populate : property name that right move will use in its return json)
{"title": extract from “summary” field in response
,"image”: from propertyImages take order 0 and transform it to the <a class as required in the provided example
,"types”: picked from propertySubType
,"size”: displaySize (note there is no space and case and spacing on Sq Ft)
,"sale_price":price”.”amount"
"link”: full url using "propertyUrl"
,"lat_lon”:constructed from "location"
"isProperty":true,
"firstDetectedOnTimestamp”: derived from "firstVisibleDate"
,"firstDetectedOn":derived from "firstVisibleDate"
,"lastSeenOn":"18-Aug-2018 11:49:45”,
"deleted":"false”,
"source”:”rightmove”/“zoopla”,
"lat":constructed from "location"
,"lng":constructed from "location"
,"area”:From area used to fetch this data from searchAreas
,"sizeInSqFt":"14945"}