Arch Systems, LLC

Compensation

Full Description

Summary Arch Systems is hiring a hands-on Executive AI Solutions Architect/Engineer to design, build, and deliver secure, production-grade AI systems for federal civilian missions (HHS, DHS, USDA, NOAA, IRS). This is a player-coach role: you will code core components yourself, lead end-to-end solutioning (discovery → design → delivery → ATO), and present solutions to federal stakeholders, clearly articulating mission impact, benefits, and ROI. Core Responsibilities • Hands-on engineering (player-coach): Write production Python; build FastAPI services; implement RAG pipelines (embeddings + vector DB); fine-tune/evaluate models; containerize (Docker) and deploy to Kubernetes with Terraform and GitHub/GitLab CI. • Time in code: Spend ≥50% of build phases actively coding, pairing, and conducting design/code reviews. • End-to-end solutioning: Lead discovery, fit-gap, and make/buy; define requirements and acceptance criteria; produce reference architectures (data → model → serving → security → observability) with trade-off analyses (cost/latency/accuracy/privacy). • Executive-ready presentations: Build decks and demos for federal audiences; clearly describe benefits and mission impact with KPIs (cycle-time reduction, precision/recall, cost per query, latency, compliance posture). • Capture & orals support: Contribute to RFIs/RFPs, technical volumes, and orals; handle Q&A with mixed technical/non-technical stakeholders. • MLOps implementation: Stand up MLflow/Databricks/SageMaker/Azure ML; manage model registry, feature store, CI/CD, gated promotions, canary/rollback; automate drift/bias/robustness monitoring and alerts. • Data engineering & governance: Build secure ingestion/curation pipelines; implement PII/PHI controls (de-identification, tokenization), data quality checks, catalogs/lineage, RBAC/least privilege, and data contracts. • GenAI/RAG safety & quality: Engineer prompt flows, grounding, guardrails, and policy enforcement; define offline/online evaluation with golden sets and human-in-the-loop; monitor factuality, relevance, toxicity; optimize context and token/latency budgets. • Security, compliance, ATO by design: Embed NIST 800-53/37 (RMF), 800-171, FedRAMP controls; implement authN/Z, encryption, secrets, logging/auditing, boundary protections; contribute to SSPs, POA&Ms, continuous monitoring; coordinate with ISSO/3PAO. • Observability & SRE: Instrument with logs/metrics/traces (OpenTelemetry); set SLIs/SLOs; build dashboards/alerts; run game days and blameless post-mortems; design HA/DR patterns to reduce MTTR and improve availability. • Performance & cost management: Model TCO; track cloud spend and cost/query; improve with right-sizing, autoscaling, caching, batching, distillation/quantization; present ROI trade-offs to stakeholders. • Reusable accelerators: Publish reference implementations, IaC modules, pipeline/evaluation templates, and security baselines; drive adoption and measure reuse impact across programs. • Delivery leadership & mentorship: Lead cross-functional agile teams; groom technical backlogs; ensure DoD (“definition of done”) includes tests, docs, security scans, and performance baselines; mentor engineers via pairing and reviews. • Documentation & change management: Maintain runbooks, ADRs, data contracts, and user guides meeting IV&V standards; manage versioning, release notes, stakeholder sign-offs; ensure Section 508 for end-user UIs/content. Minimum Qualifications (Must-Have) • 10+ years software/ML engineering; 5+ years architecting and shipping enterprise AI/ML solutions. • Recent, hands-on delivery with Python, PyTorch or TensorFlow, API development (FastAPI), containers (Docker), and microservices on Kubernetes. • Demonstrated GenAI/RAG delivery: embeddings, vector databases (FAISS/Milvus/pgvector), prompt orchestration/guardrails, evaluation and optimization. • Proven solutioning experience: leading discovery, estimating, architecture decisions, risk/cost trade-offs, and creating exec-ready artifacts that communicate impact and benefits. • Implemented MLOps on AWS and/or Azure using MLflow/Databricks/SageMaker/Azure ML; CI/CD for models and data pipelines; model registry; automated testing. • Federal delivery with working knowledge of FISMA/FedRAMP/RMF and ATO processes; ability to brief mixed technical/non-technical federal audiences. • Excellent communication/storytelling with quantitative framing (KPIs, SLAs/SLOs, ROI). Preferred Qualifications • Public Trust or clearance eligibility; prior support within HHS, DHS, USDA, NOAA, or IRS. • Certifications: AWS Solutions Architect, Azure Solutions Architect/DP-203/DP-100; security/architecture (CISSP, CCSP, TOGAF) a plus. • Experience with model risk management, adversarial testing/red teaming, and Section 508 evaluation practices. • Technical oversight is a plus (while remaining hands-on).