Databricks Platform Engineer
Fully Remote - US
Databricks Platform Engineer
Full Time • Fully Remote - US
Benefits:
- 401(k) matching
- Dental insurance
- Health insurance
- Paid time off
- Parental leave
- Vision insurance
Role Description:
The Databricks Platform Engineer will build, maintain, and optimize the enterprise data lakehouse infrastructure. The ideal candidate will have strong DevOps and data engineering skills, with expertise in automating Databricks platform operations, ensuring reliability, and enabling data teams to efficiently leverage the platform. You will be responsible for platform provisioning, monitoring, cost optimization, and providing technical support for Databricks
workloads.
Key Responsibilities include
- Build and maintain Databricks workspace infrastructure across development, staging, and production environments
- Design, implement, and maintain Databricks platforms in cloud environments (Azure, AWS, or GCP), ensuring scalability, reliability, and cost-efficiency. Manage and optimize Databricks clusters, including autoscaling, resource allocation, monitoring, and troubleshooting.
- Oversee the deployment and configuration of Databricks workspaces, including user management, access controls, and workspace policies.
- Collaborate with infrastructure and DevOps teams to implement CI/CD pipelines for deploying Databricks workflows and notebooks. Monitor and manage performance, logging, and system health of the Databricks platform. Implement robust data security and governance measures in compliance with organizational policies and standards.
- Work with cross-functional teams to troubleshoot and resolve platform-related issues.
- Automate repetitive tasks, such as provisioning, upgrades, and scaling, using Infrastructure-as-Code (IaC) tools like Terraform or CloudFormation.
- Enable data engineers and data scientists by providing support for integrating Databricks with various data sources and tools.
- Stay up-to-date with the latest Databricks features, updates, and best practices to ensure the platform meets evolving business needs.
Minimum Skills:
- 5+ years of experience in platform engineering, cloud infrastructure, or IT operations, with 2+ years of hands-on experience managing Databricks environments.
- Strong knowledge of Databricks administration, including cluster setup, workspace management, and job orchestration.
- Proficiency in Apache Spark for distributed data processing.
- Hands-on experience with cloud platforms (Azure Government, AWS GovCloud, or GCP Public Sector).
- Proficiency in scripting and automation with Python, Bash, or PowerShell.
- Experience with Infrastructure-as-Code (IaC) tools like Terraform, CloudFormation, or ARM templates.
- Solid understanding of cloud networking concepts, such as VPCs, subnets, security groups, and private endpoints.
- Familiarity with federal compliance frameworks, including FedRAMP, FISMA, NIST 800-53, and CMMC.
- Experience implementing data security best practices, such as RBAC, encryption (at rest and in transit), and audit logging.
- Ability to set up and use monitoring and logging tools (e.g., Datadog, AWS CloudWatch, or Azure Monitor) to ensure platform stability.
- Proficiency in diagnosing and resolving cluster performance, job failures, and other technical issues.
Preferred Skills
- Databricks Certified Associate or Platform Administrator certification
- Knowledge of Unity Catalog administration and data governance implementation
- Experience with cost optimization strategies for Databricks, including efficient resource allocation, autoscaling, and workload management.
- Advanced knowledge of performance tuning for Databricks clusters and Spark jobs.
- Implementation of fine-grained access controls and data masking to comply with federal data protection standards.
- Experience implementing single sign-on (SSO), multi-factor authentication (MFA), and data retention policies for secure environments.
- Hands-on experience building automation scripts for platform provisioning, monitoring, and maintenance.
- Familiarity with automation frameworks and tools that integrate with cloud platforms (e.g., AWS Lambda, Azure Logic Apps, Google Cloud Functions).
- Knowledge of ETL/ELT pipelines and experience integrating Databricks with data warehousing solutions like Redshift, Snowflake, or BigQuery.
- Familiarity with real-time data streaming technologies, such as Kafka, AWS Kinesis, or Azure Event Hubs
- Federal cloud certifications such as AWS Certified Solutions Architect – Associate/Professional, Azure Administrator Associate, or Google Cloud Associate Cloud Engineer.
- Public Trust (or higher) clearance eligibility; prior federal client experience.
This is a remote position.
Who We Are
We are a leading System Integrator and at the forefront of new exponential technologies – including Artificial Intelligence, RPA, and the Internet of Things. We help clients design the future-state of their business using Digital Cloud Solutions, and then we make IT real.
Agility Technologies Inc. is 8(a) SBA, GSA Schedule 70 registered, minority-owned a small business specializing in cloud implementation for Federal Agencies and Commercial customers. We are experts at IT Strategic Planning and Digital Transformation, Lean EA and BPM, IV&V, Application Development, and Program Management.
The ways companies can learn from and adjust to today’s crisis will deeply influence their performance in tomorrow’s changing world, providing the opportunity to retain greater agility as well as closer ties with customers, employees, and suppliers. Now is the time to move forward with Digital initiatives by adopting Next-Gen technologies.
OUR VALUES
Agility Technologies as the name suggests is an Agile organization that has created a cohesive community with a common culture. Cultural norms are reinforced through positive peer behavior and influence in a high-trust environment. We have creative problem solvers with critical thinking skills who are resilient and battle-tested. Our team members are always invited to share their ideas with the group through Brown bag sessions for peak performance to occur. We believe to minimize risk and succeed in a constantly evolving environment, the best way is to embrace uncertainty and be the quickest and most productive in trying new things.
(if you already have a resume on Indeed)