HubSpot

Lead technical strategy for observability platform focusing on distributed tracing, metrics, and logging at scale. | Extensive experience building observability or telemetry tooling for internal engineering teams at scale with deep expertise in OpenTelemetry and distributed systems architecture. | POS-5690 About the Team The Observability team owns the internal platform that gives every HubSpot engineer real visibility into how their systems behave in production. We build and operate the distributed tracing, metrics, alerting, and logging infrastructure that spans hundreds of microservices, billions of daily events, and thousands of engineers who depend on that signal to ship reliably. We are now investing in the next generation of this platform. As HubSpot deploys AI agents and ML-powered features across the product, the team is building the tracing and telemetry primitives that make it possible to understand, debug, and trust what those systems are doing in production. This is greenfield, technically interesting work at a scale few companies operate at and we are looking for a Principal Engineer to help lead it. About the Role We are seeking a Principal Software Engineer to be the technical anchor for HubSpot’s Observability platform. This role sits at the intersection of large-scale distributed systems, developer platform design, and AI observability. A big part of this role is working horizontally across a large engineering org and setting patterns and standards that make it easier for teams to instrument, alert on, and reason about their services. You will also shape how we trace and understand our growing fleet of AI agents and ML systems in production: a technically distinct and increasingly critical problem. Key Expectations Observability Platform Architecture: Define the patterns and evolution of HubSpot’s core telemetry platform — distributed tracing, metrics, and structured logging — at a scale that spans hundreds of services and billions of daily events. Set the standards for how instrumentation is done across a large, polyglot engineering organization. AI & Agentic Observability: Lead the technical strategy for tracing and understanding AI agents and ML-powered systems in production. Define the primitives, telemetry standards, and debugging workflows that help product engineers understand what their models and agents are doing — and build trust in those systems over time. This is greenfield and consequential work. High-Cardinality, High-Throughput Systems: Architect telemetry pipelines and storage systems that handle high-cardinality data at high throughput without blowing up cost or query latency. Make principled tradeoffs between sampling, fidelity, retention, and developer ergonomics. Hands-on, High-Leverage Builder: Ship production code. Lead design reviews and take high-impact initiatives end-to-end, from prototype to production system at scale. Stay close to the systems you build and be the person who can debug the hardest problems when they surface. Developer Experience & Adoption: Design the instrumentation APIs and libraries that product engineers reach for, making correct observability the path of least resistance. Drive OpenTelemetry adoption across a large, polyglot codebase. Build the tooling that turns raw telemetry into actionable signal for teams operating at speed. Production Intelligence & Reliability Patterns: Define patterns for SLO/SLI design, alerting philosophy, and how teams graduate from reactive to proactive incident response. Push for simplicity in a domain that wants to get complicated, and consistency where tooling can drift across a large organization. Technical Leadership & Influence: Partner with infrastructure, platform, and product engineering teams to understand their signal gaps and close them. Influence technical strategy alongside engineering leadership, translating observability constraints and opportunities into product and operational decisions. Mentor senior engineers and tech leads, driving thoughtful design decisions and capturing learnings from major incidents and large-scale migrations. What You Bring Platform-Builder Experience: Proven experience building observability or telemetry tooling for internal engineering teams, rather than simply consuming it. You understand how to architect developer platforms that serve thousands of engineers across a large organization, backed by deep operational instincts and hard-earned expertise. Telemetry Systems Depth: Deep expertise navigating trade-offs in telemetry pipeline design across high-cardinality data, dynamic sampling, query latency, retention economics, and data fidelity. Strong technical fluency with OpenTelemetry, distributed tracing engines, metric backends, and large-scale log ingestion infrastructure. Incident Automation & Operational Excellence: Proven track record linking telemetry signals directly to automated operational workflows. You have designed architecture for real-time telemetry triggers that power automated remediation, dynamic runbooks, or AI-assisted root-cause diagnosis across microservices environments. Why This Role, Why Now HubSpot is scaling fast — more engineers, more microservices, more AI systems running in production — and the Observability platform is at an inflection point. The foundations are solid, but the next phase requires a different kind of investment: rethinking cardinality economics, making OpenTelemetry the default across a large polyglot org, and building an entirely new layer of AI tracing that doesn’t yet exist. This Principal Engineer will have a direct line of sight from the architecture they design to the engineering outcomes we measure: incident response times, adoption rates, developer satisfaction, and the trust that product teams place in their production signal. The AI observability layer in particular is greenfield — there is no playbook to follow, which is exactly what makes this the right moment for the right person. If you want to build the platform that helps thousands of engineers understand what their systems are doing — including systems powered by AI that are genuinely hard to see inside — this is the role. Pay & Benefits The cash compensation below includes base salary, on-target commission for employees in eligible roles, and annual bonus targets under HubSpot’s bonus plan for eligible roles. In addition to cash compensation, some roles are eligible to participate in HubSpot’s equity plan to receive restricted stock units (RSUs). Some roles may also be eligible for overtime pay. Individual compensation packages are tailored to your skills, experience, qualifications, and other job-related reasons. This resource will help guide how we recommend thinking about the range you see. Learn more about HubSpot’s compensation philosophy. Benefits are also an important piece of your total compensation package. Explore the benefits and perks HubSpot offers to help employees grow better. At HubSpot, fair compensation practices aren’t just about checking off the box for legal compliance. It’s about living out our value of transparency with our employees, candidates, and community. Annual Cash Compensation Range: $313,800—$502,080 USD At HubSpot, we value both flexibility and connection. Whether you’re a Remote employee or work from the Office, we want you to start your journey here by building strong connections with your team and peers. If you are joining our Engineering team, you will be required to attend a regional HubSpot office for in-person onboarding. If you join our broader Product team, you’ll also attend other in-person events, such as your Product Group Summit and other gatherings, to continue building on those connections. If you require an accommodation due to travel limitations or other reasons, please inform your recruiter during the hiring process. We are committed to supporting candidates who may need alternative arrangements About HubSpot HubSpot (NYSE: HUBS) is an AI-powered customer platform with all the software, integrations, and resources customers need to connect marketing, sales, and service. HubSpot's connected platform enables businesses to grow faster by focusing on what matters most: customers. At HubSpot, bold is our baseline. Our employees around the globe move fast, stay customer-obsessed, and win together. Our culture is grounded in four commitments: Solve for the Customer, Be Bold, Learn Fast, Align, Adapt & Go!, and Deliver with HEART. These commitments shape how we work, lead, and grow. We’re building a company where people can do their best work. We focus on brilliant work, not badge swipes. By combining clarity, ownership, and trust, we create space for big thinking and meaningful progress. And we know that when our employees grow, our customers do too. Recognized globally for our award-winning culture by Comparably, Glassdoor, Fortune, and more, HubSpot is headquartered in Cambridge, MA, with employees and offices around the world. Explore more: HubSpot Careers Life at HubSpot on Instagram If you need accommodations or assistance due to a disability, please reach out to us using this form. Massachusetts Applicants: It is unlawful in Massachusetts to require or administer a lie detector test as a condition of employment or continued employment. An employer who violates this law shall be subject to criminal penalties and civil liability. Germany Applicants: (m/f/d) - link to HubSpot's Career Diversity page here. India Applicants: link to HubSpot India's equal opportunity policy here. HubSpot may use AI to help screen or assess candidates, but all hiring decisions are always human. More information can be found here. By submitting your application, you agree that HubSpot may collect your personal data for recruiting, global organization planning, and related purposes. We may use CLEAR ID Verification during the hiring process to confirm your identity and help maintain a safe, secure, and trusted experience for all candidates. Refer to HubSpot's Recruiting Privacy Notice for details on data processing and your rights.

Lead security program initiatives and SOX compliance efforts in a SaaS environment. | 12+ years technical program management in security/software engineering, strong technical and compliance knowledge, AI fluency. | POS- 29555 About You: You’re a highly technical, execution-driven program leader who thrives working alongside security engineers and product leaders to ship meaningful security improvements at scale. You bring structure to complex, fast-moving initiatives without slowing them down, and you earn trust with engineers and senior leaders alike by understanding the work deeply enough to drive it forward. You shape platform direction, influence executive decisions, and connect security program outcomes to long-term business impact. You’re comfortable operating autonomously in ambiguous environments, know how to navigate organizational complexity, and use AI as a genuine force multiplier in your daily work. About Us: The HubSpot Security team is dedicated to helping businesses grow better by safeguarding the systems and data that power our global CRM platform. With more than 125,000 customers in over 100 countries, HubSpot’s security posture is essential to our mission and a competitive differentiator. As part of the Security Governance and Risk team, you’ll: Partner closely with security engineering teams to drive the Security Product Line’s highest-priority initiatives from planning through delivery. Operate at the intersection of security, engineering, and governance, keeping complex multi-team programs moving and unblocking what needs to get done. Shape how the Security Governance group operates at scale, building the operating rhythms, frameworks, and tooling that let security teams do their best work. Join a culture that values transparency, learning, and authenticity. In this role you will… Security Engineering Partnership (Core Focus) Serve as an embedded TPM for security engineering teams, owning program execution for the Security Product Line’s most complex and strategically important initiatives. Drive end-to-end delivery of large-scale security programs spanning detection and response, identity and access, infrastructure security, application security, and more. Partner with security engineering leads and product managers to define roadmaps, sequence work, manage dependencies to keep initiatives on track. Navigate ambiguity and organizational complexity independently, escalating the right things to the right people and clearing blockers before they become delays.Champion scalable processes and durable systems within the security organization, raising the operational bar across the product line. Own portfolio-level KPIs, building reporting that connects program delivery to customer, reliability, and business outcomes. Serve as a thought leader within the TPM function, sharing strategies,approaches, and AI enabled workflows that elevate how the whole team works. SOX Compliance for UBB (Near-Term Priority) In the near term, this role will focus on supporting SOX compliance for HubSpot’s Usage-Based Billing features. The goal is to build a streamlined, repeatable process for bringing UBB features into SOX compliance, at which point this role hands off to a more sustainable operating model and shifts fully into security engineering TPM support. Partner with FinTech and FinOps program management to coordinate SOX compliance work across Engineering, Finance, and Security, keeping the program moving and stakeholders aligned. Drive implementation of SOX-compliant technical controls, partnering closely with engineering teams to sequence the work, remove blockers, and get features across the line. Ensure UBB features are integrated into the right SOX control frameworks and that engineering teams understand compliance requirements well enough to build compliantly from the start. Build a repeatable, scalable playbook so new UBB features can be evaluated, instrumented, and brought into SOX compliance without starting from scratch each time. AI-Powered Execution Actively use AI tools (e.g., Claude, ChatGPT, or similar) to accelerate program planning, documentation, risk analysis, and stakeholder communications. Model AI-fluent working habits for the team, sharing workflows and prompting strategies that multiply output quality and speed. Identify where AI materially changes how security engineering programs are planned, executed, or measured, and translate those opportunities into program strategy and investment priorities. We are looking for people who have… 12+ years of technical program management experience, with a significant portion embedded with security or software engineering teams in a SaaS or cloud environment. Deep enough technical fluency to earn credibility with security engineers, understand the work, and ask the right questions without needing to be a practitioner. Experience partnering with senior security engineering leaders to shape platform direction, technical standards, and execution guardrails, and the ability to connect security program outcomes to long-term business leverage and security risk. Hands-on experience leading security programs across domains such as detection and response, identity and access management, infrastructure security, application security, and more. The ability to build durable operating systems: program structures, escalation paths, reporting cadences, and tooling that outlast individual initiatives. Familiarity with compliance security frameworks and standards such as SOX, ISO 27001, NIST, SOC 2, or MITRE ATT&CK sufficient to navigate security conversations and program requirements. Demonstrated AI fluency, with a habit of using AI tools to improve the quality, speed, and consistency of program management work. You don’t just know these tools exist; you use them daily and have strong opinions on how they fit into a TPM’s workflow. Experience as a force multiplier for a TPM team, including developing methodologies, mentoring peers, and raising the bar on program quality. Excellent written and verbal communication skills, with the ability to translate complex technical and operational topics for diverse audiences. Nice to have: Hands-on experience working with security engineering teams on major platform or infrastructure security initiatives. Familiarity with SaaS architecture and cloud infrastructure (AWS/GCP) at a depth that helps you understand security engineering trade-offs. Certification(s) such as PMP, CISM, CISSP, or CISA. Prior experience with Usage-Based Billing or revenue recognition controls in a SaaS context. Hands-on experience with Asana, GitHub, or similar tools to manage large-scale security programs. Experience integrating AI tools into security or engineering workflows, such as using LLMs for documentation, risk summarization, or program reporting. Pay & Benefits The cash compensation below includes base salary, on-target commission for employees in eligible roles, and annual bonus targets under HubSpot’s bonus plan for eligible roles. In addition to cash compensation, some roles are eligible to participate in HubSpot’s equity plan to receive restricted stock units (RSUs). Some roles may also be eligible for overtime pay. Individual compensation packages are tailored to your skills, experience, qualifications, and other job-related reasons. This resource will help guide how we recommend thinking about the range you see. Learn more about HubSpot’s compensation philosophy. Benefits are also an important piece of your total compensation package. Explore the benefits and perks HubSpot offers to help employees grow better. At HubSpot, fair compensation practices aren’t just about checking off the box for legal compliance. It’s about living out our value of transparency with our employees, candidates, and community. Annual Cash Compensation Range: $172,700—$246,700 USD We know the confidence gap and impostor syndrome can get in the way of meeting spectacular candidates, so please don’t hesitate to apply — we’d love to hear from you. If you need accommodations or assistance due to a disability, please reach out to us using this form. At HubSpot, we value both flexibility and connection. Whether you’re a Remote employee or work from the Office, we want you to start your journey here by building strong connections with your team and peers. If you are joining our Engineering team, you will be required to attend a regional HubSpot office for in-person onboarding. If you join our broader Product team, you’ll also attend other in-person events, such as your Product Group Summit and other gatherings, to continue building on those connections. If you require an accommodation due to travel limitations or other reasons, please inform your recruiter during the hiring process. We are committed to supporting candidates who may need alternative arrangements Massachusetts Applicants: It is unlawful in Massachusetts to require or administer a lie detector test as a condition of employment or continued employment. An employer who violates this law shall be subject to criminal penalties and civil liability. Germany Applicants: (m/f/d) - link to HubSpot's Career Diversity page here. India Applicants: link to HubSpot India's equal opportunity policy here. About HubSpot HubSpot (NYSE: HUBS) is an AI-powered customer platform with all the software, integrations, and resources customers need to connect marketing, sales, and service. HubSpot's connected platform enables businesses to grow faster by focusing on what matters most: customers. At HubSpot, bold is our baseline. Our employees around the globe move fast, stay customer-obsessed, and win together. Our culture is grounded in four commitments: Solve for the Customer, Be Bold, Learn Fast, Align, Adapt & Go!, and Deliver with HEART. These commitments shape how we work, lead, and grow. We’re building a company where people can do their best work. We focus on brilliant work, not badge swipes. By combining clarity, ownership, and trust, we create space for big thinking and meaningful progress. And we know that when our employees grow, our customers do too. Recognized globally for our award-winning culture by Comparably, Glassdoor, Fortune, and more, HubSpot is headquartered in Cambridge, MA, with employees and offices around the world. Explore more: HubSpot Careers Life at HubSpot on Instagram HubSpot may use AI to help screen or assess candidates, but all hiring decisions are always human. More information can be found here. By submitting your application, you agree that HubSpot may collect your personal data for recruiting, global organization planning, and related purposes. We may use CLEAR ID Verification during the hiring process to confirm your identity and help maintain a safe, secure, and trusted experience for all candidates. Refer to HubSpot's Recruiting Privacy Notice for details on data processing and your rights.