MongoDB + Elasticsearch Sync Tool

$20000-50000 CAD

Closed

Posted

12 days ago

$20000-50000 CAD

Paid on delivery

We are looking for an expert in MongoDB + ElasticSearch infrastructure. We are looking to build from the ground up, in the cloud (AWS), a very complex infra with 500M Person object, 50M Company, and dozens more challenges. The dataset will be always evoling, with hundreds of thousands of updates a day on every object etc. Please see the screenshot that explain the technical setup wanted: [[login to view URL]]([login to view URL]) The final goal is to have a full blown searchable database, with list feature, and respond time always staying below a clean 100ms. The main DB will be Mongo, that needs to be optimized (shard, cluster config, index...) properly. On top of that DB will live Elastisearch, which will require a near real-time and bulletproof syncing between the two. The server budget expenses is 5-10k$ which should be enough to sustain such a setup. You will have to: Comment and detail what you think about our current thought-out infra strategy (see attached) Make large scale testing by generating hundreds of millions of fake elements Perform load tests on the infra you built Build a primary API to perform deep searches All of that in the cloud on AWS - noting that if ElasticSearch is for example not available, we can use OpenSearch or other. What matters is the final results, and to try to stick as much as possible with the proposed diagram. PLEASE: send over your thoughts about the diagram, as well as previous experience dealing with such challenges etc, and if we like your approach, we will be in touch!

MongoDB

Elasticsearch

Database Administration

Database Design

Amazon Web Services

Project ID: 38097816

About the project

41 proposals

Remote project

Active 8 days ago

Looking to make some money?

Email address

Benefits of bidding on Freelancer

Set your budget and timeframe

Get paid for your work

Outline your proposal

It's free to sign up and bid on jobs

41 freelancers are bidding on average $33,703 CAD for this job

@OutsourceMan

As an accomplished web and app developer with over 18 years of experience, I have tackled complex projects similar to what you need for your MongoDB + Elasticsearch Sync Tool. My expertise in Database Administration and Design, coupled with my deep understanding of MongoDB, makes me the perfect fit for this project. Moreover, I have previously set up scalable databases, managed undesr the pressure of constant changes and heavy loads; precisely the kind that your project calls for. I understand that optimizing your database is vital. I will(in conjunction with my team) adeptly deploy necessary techniques such as sharding, cluster configuration, and indexing to enhance MongoDB's abilities. I’m also proficient in Elasticsearch which will be seamlessly integrated ensuring bulletproof syncing between the two databases. Additionally, my past performance entails conducting large-scale testing on critical systems such as your sizable dataset. Using advanced methods like generating hundreds of millions of fake elements and load testing, I will ensure your infrastructure is robust enough to handle the dataset effectively while maintaining optimum response times below 100ms. Overall, choosing me guarantees you a diligent expert who delivers comprehensive solutions with finesse and meets every aspect of your project's requirements.

$35,000 CAD in 7 days

5.0

(18 reviews)

6.1

@iPangram

Hello There, I hope this message finds you well. I have carefully reviewed your requirement for MongoDB + Elasticsearch Sync Tool. We have extensive experience dealing with complex infrastructure challenges, including MongoDB and Elasticsearch integration. Our team is committed to delivering a high-quality solution that exceeds your expectations.  We appreciate the thought-out infra strategy you've provided. Our team will thoroughly analyze the diagram and provide detailed feedback on its strengths and areas for improvement. We will optimize MongoDB for performance and scalability, including sharding, clustering configuration, and indexing.  We will conduct extensive large-scale testing by generating hundreds of millions of fake elements to simulate real-world scenarios. This testing will help validate the performance and reliability of the infrastructure under high loads. For more understanding of your requirement please answer some questions: • Are there any specific performance metrics or thresholds that you would like us to prioritize during load testing? • Do you have any preferences for the programming language or framework to be used for developing the primary API? • Are there any specific security requirements or compliance standards that need to be addressed in the infrastructure design? Thanks, Jay

$40,000 CAD in 7 days

4.9

(7 reviews)

6.5

@manojmalviya93

As an AWS Certified Engineer with a particular focus on Azure deployments, I feel confident in saying that I am well-equipped to handle your project. My experience specifically includes proficiency in MongoDB Ops Manager, which means I know how to deploy, monitor, backup and scale MongoDB effectively on varying infrastructures. Additionally, my skillset extends to Full Stack Development and Web Scraping - meaning not only do I know the ins and outs of databases but also how they interplay with other aspects of your project. I certify that I have over 10 years of experience with using this software to create seamless connections between databases. This aligns perfectly with your need for real-time syncing between MongoDB (your primary database) and ElasticSearch (the search engine layered atop it). Notably, given my years of service in this field and my dedication to staying up-to-date with emerging technologies like OpenSearch,I am confident in my ability to navigate any potential kinks that may arise during the project.

$50,000 CAD in 7 days

4.9

(35 reviews)

5.7

@toseef3

Having successfully tackled numerous challenging projects over my decade-long career, I'm well-versed in designing and optimizing database infrastructures to ensure seamless operations. As evident from your complex setup—MongoDB as the main database with Elasticsearch running on top—I am comfortable working on such demanding environments. In fact, I understand and appreciate the importance of shard, cluster configuration, and indexing in MongoDB for optimal functioning. To validate and evaluate the capabilities of the infrastructure, I propose using large scale testing scenarios by generating hundreds of millions of fake data followed by rigorous load tests. My extensive experience navigating AWS will be instrumental as we undertake this journey together. I'm confident that my skills are aligned with your project requirements, from building a reliable sync tool to constructing a robust primary API for deep search capabilities. Lastly, Peek International has always held client satisfaction at its core. I prioritize understanding your objectives and delivering results that align with them perfectly. By choosing me for the job, you're not only hiring a seasoned expert but also someone who'll become a partner in enhancing your business's efficiency and productivity. Let's proceed with transforming your vision into reality!

$35,000 CAD in 50 days

4.7

(3 reviews)

4.8

@Giovanni704

Hi there! As I have read, I see you need help building a complex MongoDB + ElasticSearch infrastructure on AWS, capable of handling a large dataset with evolving data and high update rates, and I could help you! I'm a professional Database Administrator and Analytics Engineer with more than 10 years of experience and more than 300 projects done here on this platform. I offer unlimited revisions and I don't require any payment in front. I always meet delivery deadlines working until I get the desired results. I know all the most used softwares to do this project specifically, I could even give you some other options in case you think is better to use other tools. Send me a message so I can show you my previous work and we can discuss more. Respectfully, Giovanni.

$20,000 CAD in 1 day

5.0

(2 reviews)

4.1

@heliosdhaka

Hi There, With over 7 years of robust experience in software development and digital marketing, I firmly believe my team and I are the perfect fit for your project. We specialize in various core stacks such as Python, JavaScript and ElasticSearch which aligns perfectly with your requirements. My team is well-equipped to perform large-scale testing generating millions of fake elements and conducting extensive load tests on the infrastructures we've built. We possess significant expertise in AWS, composing appropriate indices for optimized performance in MongoDB, ensuring Mongo cluster is well-configured with proper shard distribution which results in better response times. As far as Elasticsearch goes, we have extensive experience in building near real-time syncing setups and designing bulletproof architectures. Our services are not limited to just backend development; we can also develop primary APIs proficiently for deep searches. Hence, by entrusting this project to us you'll not just get an optimized infrastructure but also a cost-effective one within your budget. Let's discuss further to strategize the best possible way to approach your dynamic yet ambitious project. We look forward to impressing you with the value we can bring to your organization through our skills and flexible mindset. Thanks Anamul

$35,000 CAD in 20 days

5.0

(6 reviews)

4.0

@revival786

Greetings! I'm a professional in providing high quality MongoDB + Elasticsearch Sync Tool creating service. I am a top-rated freelancer with about 15 years of experience, having served about 700 satisfied clients. Please message me to check my portfolio. Thanks, Revival

$20,000 CAD in 30 days

5.0

(2 reviews)

3.0

@managersingh

Hi Hope you are doing well, READY TO START NOW As an experienced Full-Stack developer having 8+ years of experience, I am excited to work with you on your project. I have read your job description carefully and understood. I have 8+ years of experience in Web design and development. I can create custom CRM, Laravel, ERP, Custom Plugin, Wix, WordPress, Shopify, MySQL, JavaScript, HTML, Woo-commerce Integration, jQuery, Third Party API integration, Bug Fixing, Troubleshooting, Malware Cleaning, Site Migration. I would appreciate if you could initiate the conversation by sending me a text in the chat box, as I have a few queries regarding the project. Thanks

$35,000 CAD in 7 days

5.0

(1 review)

1.6

@istackgroup

With over a decade of experience as a full stack developer, I am equipped with the skills and knowledge necessary to tackle your MongoDB+Elasticsearch infrastructural needs. My vast experience includes working with large datasets and designing robust architectures that prioritize searchability and responsiveness while ensuring data accuracy. Having worked extensively with cloud deployment on AWS, I am well-versed in optimizing server configurations to maximize budget efficiency while achieving high-performance results. Moreover, my commitment to clients is exemplified by my track record of successfully completing over 300 projects to their satisfaction. My approach involves generating millions of simulated data points for comprehensive testing, followed by using the insights gained to create an efficient API for deep search functionality. To conclude, my ability to adapt quickly, solid communication skills, and proven history of delivering high-quality work within strict timelines makes me the ideal fit for your intricate infrastructural requirements. Let's discuss further how we can optimize your current strategy and build a powerful system that maximizes insights from your MongoDB data via Elasticsearch while being cost-effective.

$35,000 CAD in 30 days

1.2

(1 review)

4.0

@tgltech

Hi Jon, We have thoroughly reviewed your requirements and the technical setup diagram for a MongoDB + ElasticSearch infrastructure on AWS. We understand the complexity of handling large datasets and the need for optimal performance. Key Points: -Design and implement a robust MongoDB architecture with sharding and proper indexing for optimal performance. -Implement ElasticSearch for full-text search capabilities and ensure real-time data syncing with MongoDB. -Conduct extensive load testing using generated fake data to simulate hundreds of millions of records. -Develop a highly efficient primary API to facilitate deep searches across the dataset. -Ensure architecture is scalable and response times are maintained under 100ms as per your requirements. Queries: -Could you specify any preferred tools or methodologies for generating test data? -Is there a specific feature or dataset focus you foresee requiring more intensive optimization? Recommendation: Considering the scale and dynamics of your dataset, implementing additional monitoring tools might enhance system reliability and provide real-time insights into performance metrics. Ready to transform challenges into successes with our expertise in MongoDB and ElasticSearch. Regards Tapan S. P.S. Budget and timeline are subject to change as per final requirements

$37,471 CAD in 7 days

0.0

(0 reviews)

0.0

@bagelhs

With over 9 years of industry experience, I have consistently proven my expertise in architecting highly scalable and efficient infrastructures just like the one you're seeking to build. Having previously built similar MongoDB and ElasticSearch based systems on AWS for extremely large datasets, I grasp the significance of optimizing the database for performance and developing a robust synchronization mechanism between the two databases. Your proposed diagram looks promising, but let's refine it further through detailed discussion and my architecture insights. To ensure optimal performance, I'll carry out extensive load testing on the infrastructure - populating your system with hundreds of millions of fake elements and meticulously evaluating various aspects of your setup. Alongside, I can develop a powerful API for deep searches that will enable you to use all the data at your disposal in an efficient manner. Rest assured, I'll stick as closely as possible to your budget without compromising the infrastructure's performance or scalability. Thanks

$30,000 CAD in 40 days

0.0

(0 reviews)

0.0

@PauloLosinada

hello! After checking the job posting, I felt that your project was similar to one I had worked on before. I have undertaken similar projects to ensure I can provide you with satisfactory results. I can start your project right away and you would be interested in discussing the project details. Thank you for taking the time to review our letter. I look forward to hearing from you regarding further steps. thank you

$20,000 CAD in 50 days

0.0

(0 reviews)

0.0

@AgustinMaidana

Drawing from my extensive experience in AWS Cloud, database administration and specifically MongoDB, I assure you that I am more than capable with the given complex infrastructure setup. Over the years, I have become well-versed in optimizing databases by effectively using MongoDB cluster configurations, sharding, and indexing to handle high data volumes and updates. What sets me apart is my ability to adapt, demonstrated by my willingness to work with alternate options like OpenSearch if needed. Additionally, my expertise in Elasticsearch ensures that I can deliver the demanded near real-time and reliable data syncing between MongoDB and Elasticsearch. To prove this, I plan on conducting thorough load tests and generating hundreds of millions of fake elements for large-scale testing of every component at the proposed budget. Furthermore, my familiarity with web concepts including the proposed diagram will facilitate creating an API for deep searches efficiently. Building upon notches proven by relevant previous experiences similar to these challenges you've outlined. Overall, Satisfaction and consistent results are what guide my work ethic. As such, exceeding your expectations with timely response rates averaging below 100ms is without a doubt a guarantee.

$20,000 CAD in 150 days

0.0

(0 reviews)

0.0

@bsbrajmohan25

Hi, Hope you are doing well. We are much expertise group of iOS, Android, Website developers and Designers Please reviews my profile - https://www.freelancer.com/u/bsbrajmohan25 I understood the initial scope of your MongoDb with ElasticSearch Sync tool. I want to discuss more about this project in order to prepare the final concept. So let’s discuss in detail once then will make plan to start work on it. Thanks.

$20,000 CAD in 90 days

0.0

(0 reviews)

0.0

@pinkeegupta

Hi I have read your requirement and I am sure I will be able to help you. Please message me so that we can have a detailed discussion. I have 8+ years of combined experience in Web designing & development, and other software development and have expertise in PHP and its framework CodeIgniter, JS frameworks React.js & and Node.js, AWS and also Java and its framework spring boot. Please come to chat for further discussion. Regards, Anju

$35,000 CAD in 50 days

0.0

(0 reviews)

0.0

@abdelghafar

Hi there! With over 12 years of experience in software engineering on large and complex projects, I'm sure I can help you finish your project with ease. I have an extensive experience in the following areas: - Software architecture - Distributed systems - Database modeling - API design - CI/CD Some of my specialties are: • .NET 8, Web APIs, SignalR, Entity Framework Core, Dapper • Angular, TailwindCSS, ReactJS, JavaScript, TypeScript . Telerik, DevExpress, AspNet Zero • C#, C++, Rust • PostgreSQL, MS SQL Server, MongoDB, Redis • RabbitMQ, Amazon SQS, Kafka, MassTransit • Azure, AWS

$50,000 CAD in 60 days

0.0

(0 reviews)

0.0

@mjanithbandara

Hello, With over 4 years of experience in full stack development and expertise in MongoDB and Amazon Web Services, I am well-poised to build and optimize a complex infrastructure like the one you're envisioning. My familiarity with MongoDB & Elasticsearch will enable me to expertly setup your database, while making sure it remains sharded and clustered for optimal performance. My proficiency extends beyond building the infra: I have consistently produced APIs that deliver good search response-time results. As such, I am confident that I can maintain response times well below 100ms on your dataset, even with ongoing updates. Having worked konkessibly with cloud-based development (AWS), I am also responsive to quick shifts based on compatibilities of tools available. In this case, we can consider other options such as OpenSearch, provided it aligns with our aim of realizing the best outcomes for this project. Given my pedigree and attitude towards innovation, I believe I'd be a valuable addition to your team. Regards, Janith.

$22,000 CAD in 30 days

0.0

(0 reviews)

0.0

@Joshart

Hi, Choosing our services guarantees you a seamless transition to a highly efficient and scalable MongoDB + Elasticsearch infrastructure tailored to your complex needs. Here's why we're the right choice: Expertise: With extensive experience in building and optimizing cloud-based infrastructures on AWS, we understand the intricacies of handling massive datasets like yours. Our team of experts specializes in MongoDB and Elasticsearch, ensuring a robust and reliable solution. Thoughtful Infrastructure Strategy: We've carefully reviewed your proposed diagram and believe it provides a solid foundation. Performance Optimization: Our primary focus will be on optimizing MongoDB, including sharding, cluster configuration, and indexing, to ensure optimal performance and scalability. Additionally, we will implement near real-time syncing between MongoDB and Elasticsearch to provide seamless search functionality. API Development: We will develop a robust API to facilitate deep searches and provide seamless integration with your applications. Our API will be designed for performance, reliability, and ease of use, enabling you to leverage the full capabilities of your database infrastructure. There isn't enough space in this bid to elaborate more and talk about the diagram, but with me you'll have a dedicated team of experts committed to delivering a high-performance, scalable, and cost-effective MongoDB + Elasticsearch infrastructure that will drive your business forward.

$20,000 CAD in 15 days

0.0

(0 reviews)

0.0

@technosystem

With 18+ years in the IT industry and a comprehensive understanding of C#, Javascript, Python, ReactJS, among others, my team and I have tackled numerous complex projects similar to yours. We have successfully implemented scalable AWS architectures and Elasticsearch infrastructures for several clients and ensured they meet the stringent performance targets. In fact, our previous project involved managing a data set similar in size to yours by creating a performant data sync solution between MongoDB and Elasticsearch. We pride ourselves on our meticulous attention to detail. Your proposed infrastructure is sound; however, we would include an additional failover for high availability as part of our approach. To confidently evaluate its capabilities, we will generate hundreds of millions of fake elements to conduct large-scale testing and subsequent load tests on the infrastructure to ensure optimal efficiency. Our goal is to deliver not just a reliable solution but one that is also easy to navigate. To this end, we will build a primary API for you – an interface that enables seamless deep searches across your database. Your budget aligns with our expectations so we can put in every effort to meet your needs within the specified range. As repeatedly evident from our past work, client satisfaction is crucial for us, and we guarantee an impeccable end product.

$42,500 CAD in 70 days

0.0

(0 reviews)

0.0

@EIIIIII

To maintain a seamless synchronization between MongoDB and Elasticsearch, using the MongoDB connector for Elasticsearch is advisable. This connector captures changes in MongoDB in real-time via the oplog and pushes these changes to an Elasticsearch cluster, thus maintaining data consistency without heavy overhead. Please share more details about your current infrastructure plans or diagrams, and we can discuss more tailored strategies and optimizations. If this approach aligns with your vision, I'd be keen to explore this opportunity further.

$36,000 CAD in 29 days