Experience: 8+ years
Job Description:
We are looking for data engineers to join our team. You will use various methods to transform raw data into useful data systems. For example, you’ll create algorithms and conduct statistical analysis. Overall, you’ll strive for efficiency by aligning data systems with business goals.
To succeed in this position, you should have strong analytical skills and the ability to combine data from different sources. Data engineer skills also include familiarity with several programming languages and knowledge of machine learning methods.
Job Requirements:
● Participate in the customer’s system design meetings and collect the functional/technical requirements.
● Responsible to meet the customer expectations on real time data integrity and implement efficient solutions
● Build up data streaming pipelines.
● Skillful in ETL Data Engineering.
● Clear understanding and experience with Spark, PySpark, Hive and RDBMS architecture.
● Proficient in writing Python programs and SQL queries.
● Proficient in SQL Query tuning.
● Experienced in Shell Scripting(Unix/Linux).
● Good experience on any of the Programming Languages Python/Scala/Java.
● Using Azure cloud services and databricks.
● Build and maintain data pipelines in Spark with SQL and Python.
● Knowledge of 2 or more Cloud (Azure/AWS/GCP, etc..) technologies is additional.
● Good understanding of Kubernetes, CI/CD concepts, Apache Kafka
● Suggest and implement best practices in data integration.
● Guide the QA team in defining system integration tests as needed.
● Split the planned deliverables into tasks and assign them to the team.
● Needs to Maintain/Deploy the ETL code and follow the Agile methodology
● Needs to work on optimization wherever applicable.
● Good oral, written and presentation skills.
Preferred Qualifications:
● Degree in Computer Science, IT, or similar field; a Master’s is a plus.
● Great numerical and analytical skills.
● Technical expertise with data models, data mining, and segmentation techniques.
Key traits:
● Should have excellent communication skills.
● Should be self motivated and willing to work as part of a team.
● Should be able to collaborate and coordinate with team members onshore and offshore.
● Be a problem solver and be proactive to solve the challenges that come his way.