Senior Python Data Engineer (Databricks, Delta Lake)

1 tuần trước

Trách nhiệm công việc

• Design, develop, and maintain data pipelines using Databricks, Delta Lake, and other relevant technologies to support data analytics and machine learning initiatives.
• Manage and optimize data storage solutions, including Data Lake, to ensure high performance and scalability.
• Work with NoSQL, SQL databases such as PostgreSQL, MongoDB to handle diverse data sets and unstructured data or structured data. 
• Implement ETL/ELT processes and data modeling solutions to transform raw data into valuable insights.
• Write complex SQL queries to manage, analyze, and manipulate data across various relational databases.
• Collaborate with data scientists, analysts, and other stakeholders to gather requirements and ensure data solutions align with business needs.
• Utilize programming languages like Python for data processing and automation of workflows.
• Monitor and troubleshoot data pipelines to ensure data quality, consistency, and availability.
• Leverage data pipeline orchestration tools such as Apache Airflow or Prefect to automate and schedule data workflows.
• Continuously improve data engineering practices by adopting new tools, techniques, and best practices in the industry.

Kỹ năng và chuyên môn

Requirements:

• 4+ years of experience in data engineering or a related role.
• Hands-on experience with Databricks and Delta Lake.
• Proficient in SQL for querying and managing relational databases.
• Experience with NoSQL databases like MongoDB or Cassandra.
• Strong programming skills in Python for data processing.
• Familiarity with cloud platforms, linux.
• Experience with data pipeline orchestration tools such as Apache Airflow, Prefect, or similar.
• In-depth understanding of data warehousing concepts, ETL/ELT processes, and data modeling.
• Strong problem-solving skills with great attention to detail.

Preferred Qualifications:

• Experience with big data technologies like Apache Spark, Hadoop, ElasticSearch or Kafka.
• Familiarity with CI/CD practices and tools for data engineering