2021 - 2022
* Using Linux TPROXY to scale up L4 K8S Pods on demand and save 50%+ costs of our dev K8S cluster.
* Using my unified PostgreSQL CDC and dumping solution to handle data changes robustly, including indexing content into Elasticsearch, watching other micro services' events, and migrating the whole database with one event consumer implementation with nearly zero downtime.
* Building CDN edge computing framework to make my colleagues utilize CDN cache at edges easily.
* Improving 10x throughput on our usage of go-redis library transparently with auto pipeline technique.
2019 - 2021
* Building an internal IaC framework with Pulumi, Packer and GCP KMS.
* Building an internal Envoy XDS control plane for better traffic management compared to K8S' service.
* Building a scalable and transparent database snapshot as a service as well as a disaster recovery mechanism based on GCE snapshots and custom proxy of PostgreSQL and MongoDB. It has now become a core platform for ETL, OLAP, and even a testing environment for query optimization. It restored 1000+ database snapshots every day in 2020 with little GCP costs. Later, I enhanced the platform to real-time replication with my CDC solution in 2021.
* Building a fast cardinality estimation service with HyperMinHash to give a quick business insight.
* Building a CDN-friendly linked list service with Golang and PostgreSQL. It allows engineers to deliver content easily, including trying different content sorting algorithms. It has also become a core communication bridge between different engineer teams because of its simple API and high performance. It now inserts 10TB+ of data every day into just one n1-highmem-4 PostgreSQL instance thanks to optimization with ZSTD compression and batching on GRPC streams.
2015 - 2018
* Building a user event tracking and CDN-friendly A/B testing remote config service with Golang and Apache Pulsar. It has also become a core data source that drives other business insights and ML sources. It now collects 2TB+ of events every day. I also made it easy for our colleagues to consume these events in Apache Flink and Spark seamlessly by writing our own connectors in Java.
* Building NodeJS APIs and internal Golang GRPC/HTTP/AMQP framework with our code generation tool.