AWS Redshift and DocumentDB, as well as batch processes to import traditional CSV files. Utilized Databricks for large-scale data processing, leveraging its Spark capabilities to efficiently transform and aggregate incoming data streams. With the combined power of Databricks and AWS Lambda, ensured unparalleled data consistency, quality, and preparedness for sophisticated analytics and reporting. Utilized Databricks and Airflow to run extensive data profiling tasks, analyzing data patterns and identifying potential quality issues before they reached the Databricks Delta Lake. Established robust guardrails using the combined might of AWS Lambda, Apache Airflow and Databricks, ensuring that data
Full-time / Quan tâm đến làm việc từ xa