Lightup enables data-driven businesses to achieve their full potential. Modern businesses are dealing with unprecedented volumes of real-time data, operating in environments that are constantly changing internally and externally. To get the most out of data-driven operations, businesses need the ability to continuously measure the performance of their data-driven operations at fine granularity across the entire pipeline. Lightup is enabling businesses to continuously track vitals of their data-driven operations at the finest granularity, catch problems before their customers do, and fix them before they impact business.
The founding team brings together a unique combination of experience in big data, stream processing and rigorous statistical signal processing - crucial for building a data stability platform. We have over three decades of industry experience in multiple game changing startups and industry stalwarts like Google, GlobalFoundries and VMware. Lightup is funded by Andreessen Horowitz (a16z) and Spectrum28.
Lightup is looking for our Data Wizard (Sr. Data Engineer) who has knowledge and ability to build large scale systems and data pipelines.
Education/Experience: You have a degree in computer science or an allied major with a few years of experience. BS with 5 years experience, MS with 3 years experience, PhD with 2 years experience.
You have understanding and have worked with:
- Data integration - ability to connect different data sources to the system - Eg: Presto, SQLchemy.
- Knowledge of streaming systems like Spark, Flink or Heron
- Data mobility like Kafka, Kinesis or Pulsar
- Understand and work with SQL systems like Postgres or MySQL
- Schema discovery - implementing automatic schema inference and hooking up metastores
- Data and data schemas - SQL, JSON, XML, Protocol Buffers, Avro, etc
- Data storage - Cassandra, DynamoDB, Druid, RocksDB,
- Data Warehousing - HDFS, S3, Snowflake, Redshift ..
- AWS services - Lambda, Elastic Beanstalk
- Building and testing large scale systems
- Leverage cloud services like AWS (Kinesis, Kafka, DynamoDB, RDS, Aurora) effectively.
What you will do at Lightup:
- Architect, design and implement our scalable data pipeline.
- Handles millions of events and millions of rows
- Integrate with lots of disparate data sources both batch and streaming data
- Implement statistical models, statistical computation and ML models on data at line rate
- Run systems efficiently and with high performance.