Senior Data Engineer

Job updated about 1 month ago

Job Description

Where are we headed:

  • We’re building the foundation of decentralized knowledge graph infrastructure for web3 and developing high quality products to showcase this.
  • Our road-map is robust, and with an iterative and continuous development philosophy, we’re always building.
  • We’re a global team that values diverse perspectives and top tier talent. We’re growing internationally with the support of top tier investors and advisors.

Where do you fit in?

We are seeking a seasoned Data Engineer to play a crucial role in building and optimizing our data products and pipelines. The individual will be tasked with developing, constructing, testing, and maintaining architectures such as databases and large-scale processing systems. Utilizing modern data orchestration tools, the Data Engineer will streamline the building of streaming and batch processed data feeds, and integrate data tools into AI models and analytics tools.

Requirements

Responsibilities

Data Pipeline Development and Optimization

  • Design, construct, install, and maintain large-scale processing systems and other infrastructure.
  • Develop data set processes for data modeling, mining, and production.
  • Employ a variety of languages and tools to marry systems together.
  • Implement data flows that connect operational systems, data analytics, and data visualization systems.

Data Integration

  • Build, develop, and maintain processes that import/extract, analyze, and visualize data, making it available to analysts and other end-users.
  • Ensure systems meet business requirements and industry practices.
  • Work towards the integration of data analytics and AI models.

Data Testing and Deployment

  • Create reliable, automated processes for verifying data quality and integrity.
  • Deploy sophisticated analytics programs, machine learning, and statistical methods.
  • Utilize CI/CD methodologies to ensure smooth transitions of data products from development to production.

Data Scalability and Performance Optimization

  • Optimize SQL, dags, or other data pipeline operations as necessary and ensure data solutions are scalable.
  • Ensure data privacy and compliance with data governance and data management principles.
  • Employ strategies for managing and processing large data sets effectively and efficiently.

Data Exploration and Visualization

  • Perform exploratory data analysis and visualization for initial insights and data familiarization.
  • Employ a myriad of tools and platforms for connecting disparate data sources and visualizing complex datasets.

Skills and Qualifications

Essential

  • Must have web3 data related work experience
  • Data Processing: Proficiency in managing large datasets, ranging from hundreds of GB to TB, with billions of entries.
  • Programming: Extensive use and knowledge of SQL and Python.
  • Data Orchestration: Experience with data orchestration tools like Apache Airflow or Dagster.
  • Data Streaming: Proficiency in managing and manipulating streaming data - e.g., experience with Kafka.
  • Cloud Platforms: Hands-on experience with cloud platforms like GCP or AWS.

Desirable

  • Data Visualization: Experience with data visualization tools such as Power BI.
  • CI/CD: Familiarity with CI/CD practices and tools for data.
  • Machine Learning Integration: Experience integrating machine learning models into data pipelines.

Educational & Experience Requirements:

  • Bachelor’s degree in Computer Science, Engineering, or related field.
  • A minimum of 3 years of experience in a Data Engineering role.
  • Fluent English, both written and spoken
  • Significant experience collaborating within an engineering team and reviewing code
  • Have exceptional technical communication skills (written and verbal)

Write to Us If You:

  • Relish problem-solving and are constantly seeking out challenging coding tasks to tackle.
  • Have an interest in blockchain technology and the transformative potential it holds.
  • Are committed to working like an owner, going beyond the confines of a conventional 9-to-5 role or contractor mentality.
  • Are comfortable assuming significant responsibility and ownership, and thrive in an environment where you’re trusted to make key decisions.

1
3 years of experience required
80 ~ 200 TWD / year
100% Remote Work
Personal Invitation Link
This is your personal referral link for job invitation. You'll receive an email notification when someone applied for the position via your job link.
Share this job
People who applied for this job also applied for
Logo of the organization.
Full-time
Mid-Senior level
1
35K ~ 45K TWD / month
Logo of the organization.
Full-time
Entry level
2
Regular earnings reach NT$40,000
Logo of the organization.
Full-time
Mid-Senior level
1
60K ~ 140K TWD / month
Logo of Bluwhale.

About us

Lucid is a leading AI startup which partner with its corporate clients to boost their retention and revenue with AI-powered personaliazation.

Harnessing the power of knowledge graphs and contextual data extraction, Lucid seamlessly generates comprehnsive user insights and valuable content data in tandem. With this robust infrastructure, our customers can access a multitude of features designed to continually enhance user experience and fuel revenue growth through AI.

For more information, visit www.lucidinside.com.


Team

Avatar of the user.
Sr. Android Developer
Avatar of the user.
UI/UX Designer
Avatar of the user.
Backend Engineer
Avatar of the user.
Sr. Android Engineer
Avatar of the user.
Sr. Android Engineer
Avatar of the user.
HR

Jobs

Freelance
Mid-Senior level
1
500 ~ 1.5K TWD / hour
Save

Full-time
Mid-Senior level
1
800K ~ 2M TWD / year
Save