AWS(Hive SQL) to GCP(BigQuery SQL) using python Migrate datawarehouse from AWS(Hive) to GCP(BigQuery) Setup Prometheus on GKE (Google Kubernetes Engine) to monitor resource usage(CPU, memory of compute engine) Largitdata, Data Quality Manager Sep 2017 ~ July 2018 Monitor status of all web scraping running scripts.( Flask ) Write and maintain web scraping scripts on distributed system.( Python + Celery + RabbitMQ / Redis ) Largitdata, Web Scraping Intern Jan 2017 ~ Aug 2017 Write many web scraping scripts for various sorts of websites. Skills Languages - Python , Scala Big Data Framework - Apache Spark, Hadoop/HDFS, GCP
National Taiwan University・
電機工程學系