Data Engineer - Azure Data Factory
anonymous company - Hong Kong
new offer (23/04/2024)
job description
- Designing and implementing end-to-end ETL processes using Azure Data Factory to extract data from multiple sources, transform it to meet business requirements, and load it into a data warehouse.
- Collaborating with cross-functional teams to understand data requirements, business rules, and data quality standards to ensure accurate and consistent data delivery.
- Building and maintaining scalable and efficient ETL pipelines that handle large volumes of data while adhering to best practices for data integration and data governance.
- Developing and implementing data validation and reconciliation processes to ensure data accuracy and consistency across all data sets.
- Troubleshooting and resolving issues related to data transformation, data loading, and data quality, while proactively identifying opportunities for process optimisation and performance tuning.
- Collaborating with data architects, data scientists, and other stakeholders to understand data models and data requirements to support data exploration, analytics, and reporting initiatives.
- Documenting and maintaining technical specifications, data pipeline documentation, and data lineage to ensure data governance and regulatory compliance.
- Staying up-to-date with the latest industry trends and technologies related to Azure Data Factory and ETL to continuously improve data engineering practices and solutions.
- Bachelor's/Master's Degree in Computer science, mathematics, statistics, economics, or other quantitative fields
- Demonstrable experience with any programming or scripting language (Python/Java/Scala/Ruby etc.)
- Strong expertise in SQL and experience with data integration, data transformation, and data loading techniques using ETL tools and frameworks
- Hands-on experience with cloud platforms, preferably Microsoft Azure, including familiarity with Azure Data Lake Storage, Azure SQL Data Warehouse, and Azure Databricks
- Proficiency in programming languages such as Python, R, Powershell, for data processing, data manipulation, and data transformation
- Knowledge of data modeling concepts and experience with data modeling tools is a plus
- Excellent problem-solving skills, with the ability to analyze complex data scenarios and design efficient and scalable solutions
- Strong communication and collaborative skills, with the ability to work effectively in cross-functional teams and communicate technical concepts to both technical and non-technical stakeholders