Design, implementation and ongoing administration of Big Data infrastructure.
Responsibilities:
Design, installation, configuration and administration of Hadoop platform and maintaining the Hadoop infrastructure and operations,
Forecasting, planning, capacity arrangement and scaling of Hadoop clusters,
Monitoring of Hadoop cluster connectivity and security, conducting performance tuning of Hadoop clusters (7/24 on-call availability will be required),
Management and monitoring of Hadoop log files and file system management,
Coordinating system backups with infrastructure team for storage and rotation of backups; setting and implementing Backup and Disaster Recovery strategy,
Automation and integration of monitoring and server job processes,
Setting up and testing of Kerberos principles, administration of Key trustee server,
Management of integration with Active Directory and support for user and group management,
Working with Big Data Engineer to setup data access security groups and working with delivery teams for provisioning of users into Hadoop,
Enabling of policies to have a privilege level access to the data in HDFS as per the security policies,
Enabling of data encryption to meet the security standards
Skills:
Strong knowledge of Hadoop Architecture (HDFS), Hadoop Cluster installation, configuration, monitoring, cluster security, cluster resources management, maintenance and performance tuning,
Expert level knowledge of Hadoop components such as HDFS, Sentry, Kafka, Impala, Hue, Hive, YARN, ZooKeeper, Postgres, HBase, Flume, Scoop, Oozie etc.,
Strong knowledge of key scripting and programming languages such as MapReduce, Spark, Python, Scala and Bash,
Knowledge of relational databases, industry practices, techniques and standards,
Administration expertise and basic knowledge of memory, CPU, OS, storage, and networks,
Proficient with system upgrades, patches and maintaining compliance,
Intimate knowledge of fully integrated AD/Kerberos authentication,
Work individually, with minimal supervision, as well as in team environments,
Detail oriented, well organized work skills,
Multitasking, time and stress management,
Strong ability to identify, prioritize and solve system related problems.
Qualifications:
Bachelor’s degree in Computer Science, Engineering or related field,