Senior Engineer at Yahoo.Jan, 2023 - Present
Working at Yahoo. The greatest challenge lies in developing in response to rapid market changes, where the data catalog needs to integrate with various complex systems. Simultaneously, maintaining the highest stability and ensuring high-quality data is essential.
More responsibilities/details as below.
- Fetch providers data with various way via Java. Such as fetching data from client's API, GraphQL, FTP, S3 or GCP... etc.
- standardizing data from above to feed into data warehouse in Hadoop/Hive/HBase using Spark.
- Implement a checking system to guarantee high quality data via Java.
- Migrate part of services from on-perm(Hadoop) to cloud(AWS) to become a hybrid cloud environment.