Data Engineer

Responsibilities:
  • Detail design on end to end ETL process flow, inline and comply with the design by solution architect team
  • Design ETL operation flow, with standard operating procedure for overall operation (including all automation process)
  • Develop and perform data ingestion from multi sources with as Informatica main ETL platform, on top of Hadoop and In-memory db
  • Develop end to end ETL process, with batch or real time mode on top Informatica as the main engine
  • Perform data quality analysis, cleansing, lineage, transformation & data feed to other business systems
  • Manage and maintain data on all layer (raw, gold, Business, 360, etc.) based on data model design
  • Develop and implement data streaming and complex event processing process
  • Maintain all merchant data references
  • Develop and maintain scheduled jobs for all data layer from raw data to API layer
  • Develop and implement the operation and maintenance tool (OMT), in all ETL process.
  • Create a Technical and Functional Document based for all jobs
  • Create testing document

Requirements:
  • Experience with object-oriented/object function scripting languages: Python, Java, C++, Scala, etc.
  • Experience with data pipeline and workflow management tools
  • Familiar with Linux/Unix
  • Experience with big data tools: Hadoop, Spark, Kafka, etc.
  • Experience with relational SQL and NoSQL databases
  • Strong project management and organizational skills.
  • Strong analytic skills related to working with unstructured datasets.
  • Build processes supporting data transformation, data structures, metadata, dependency and workload management.

Want to apply later?

Type your email address below to receive a reminder

Apply to Job

ErrorRequired field
ErrorRequired field
ErrorRequired field
Error
Error
insert_drive_file
insert_drive_file