Middle/Senior Data Engineer

1 week ago

Job Responsibilities

• Design, build, and maintain efficient and reliable real-time data pipelines to support live sports scoring and analytics.
• Develop architectures and databases optimized for fast querying and real-time data processing on physical servers.
• Work with sports data feeds and APIs to capture, store, and normalize data for real-time usage.
• Collaborate with the frontend and analytics teams to implement features that rely on real-time data processing.
• Ensure system consistency and data integrity with high availability and performance.
• Monitor system performance, implement necessary adjustments, and respond to system-wide issues in a timely manner.
• Manage data storage solutions on physical infrastructure, focusing on security and data recovery.
• Utilize modern data processing technologies like Kafka and Spark, suited to streaming and real-time analytics.
• Document system architecture and maintain data flow diagrams for internal use.
• Develop strategies for integrating multiple data sources and APIs, ensuring smooth data flow and consistency across different platforms.
• Implement data crawling techniques to enhance the collection processes from various data sources and improve the breadth of data available for analytics.
• Conduct research to identify new data sources and evaluate their potential integration into the system to improve product offerings.

Job Requirements

• Bachelor's or Master’s degree in Computer Science, Engineering, or a related field.
• 3+ years of experience as a Data Engineer, with a strong background in real-time data systems.
• Proficiency in Python, Java, or similar programming languages.
• Experience with real-time data processing tools (Kafka, Spark Streaming, etc.).
• Familiarity with sports data APIs and their integration into live systems.
• Strong understanding of SQL and NoSQL databases, such as PostgreSQL and Cassandra.
• Knowledge of managing physical servers and optimizing server performance.
• Proven ability to troubleshoot and optimize data pipelines and architectures for real-time operations.
• Excellent project management skills and the ability to work collaboratively in a team environment.
• Experience in data mapping and normalization across various data sources and APIs.
• Skilled in designing and implementing data crawlers to automate data collection and enhance real-time data availability.