Experience: 3+ Years
Job Title: Web Scraping Engineer / Data Extraction Specialist - Python
About Us:
We are a dynamic and innovative team specializing in building scalable and efficient web scraping solutions for complex e-commerce platforms and diverse web environments. Our team is proficient in a wide range of technologies and advanced web scraping techniques, ensuring high-quality data extraction and delivery.
Job Summary:
We are seeking a talented Web Scraping Engineer / Data Extraction Specialist to join our growing team. The ideal candidate will have a strong background in web scraping, data extraction, and backend technologies. You will be responsible for designing, developing, and maintaining robust web scraping solutions, handling dynamic content, and overcoming anti-crawling measures.
Responsibilities:
● Develop and maintain scalable web scraping scripts using Python, JavaScript, and related frameworks (e.g., Scrapy, Selenium, Puppeteer, Beautiful Soup, Cheerio.js).
● Implement advanced web scraping techniques, including API interception, sitemap parsing, and handling dynamic content.
● Design and build data pipelines for efficient data extraction, processing, and storage.
● Manage and optimize data extraction workflows, ensuring high speed and accuracy.
● Implement anti-crawling solutions, including IP rotation, proxy management, and CAPTCHA bypassing.
● Collaborate with cross-functional teams to gather requirements and deliver complex data solutions.
● Utilize backend technologies such as Flask, FastAPI, Django, Node.js, Spring Boot, and relational databases (PostgreSQL, MySQL) for data storage and API development.
● Work with cloud platforms like Azure and leverage services such as AzureML and ADLS GEN2.
● Employ data processing techniques using libraries like NumPy and Pandas.
● Use tools like Postman, MITM, and DevTools for API testing and network traffic analysis.
● Apply machine learning and NLP techniques for data analysis and processing (e.g., sentiment analysis, content classification).
● Set up and manage server-side scraping environments.
● Monitor and troubleshoot scraping scripts to ensure optimal performance.
Required Skills:
● Strong proficiency in Python and/or JavaScript.
● Experience with web scraping libraries and frameworks (Scrapy, Selenium, Puppeteer, Beautiful Soup, Cheerio.js).
● Knowledge of backend technologies (Flask, FastAPI, Django, Node.js, Spring Boot).
● Experience with relational databases (PostgreSQL, MySQL).
● Understanding of HTTP/HTTPS protocols and API communication.
● Familiarity with cloud platforms (Azure).
● Ability to handle dynamic content and JavaScript-heavy websites.
● Experience with anti-crawling techniques (IP rotation, proxies, CAPTCHA bypassing).
● Data processing and analysis skills (NumPy, Pandas).
● Experience with API testing tools (Postman).
● Knowledge of machine learning and NLP concepts is a plus.
● Strong problem-solving and debugging skills.
● Excellent communication and collaboration skills.
Experience:
● 3+ years of professional experience in web scraping or related fields.
Education:
● Bachelor's degree in Computer Science, Engineering, or a related field.