Principal Architect, Agentic AI & Platforms Edit listing

BICP, a San Diego-based consulting firm specializing in AI, Data & Analytics, is seeking an experienced Principal Architect, Agentic AI & Platforms to join our team in an embedded capacity with one of our strategic clients in the apparel and retail industry. This role is in San Diego and requires onsite Monday through Friday.

The organization is experiencing significant demand for Agentic AI solutions across the business and is now building the platform, engineering practices and internal capabilities required to move from pilots to secure, scalable production delivery.

This is a hybrid architecture and technical leadership role. You will define the shared AI platform while also providing hands-on architectural direction for the first agentic products built on top of it. You will evaluate existing and emerging platforms, establish reusable patterns, guide technical implementation and mentor experienced engineers and data scientists who are developing agentic AI expertise.

**Key Responsibilities**

*Define the Enterprise AI Platform*

* Establish the target architecture and technical roadmap for an enterprise AI and agentic AI platform.
* Design reusable platform capabilities that support multiple AI products, business functions and development teams.
* Define how product teams access models, enterprise data, tools, APIs, evaluation services and deployment environments.
* Establish a secure, scalable "paved road" that allows teams to move quickly without independently rebuilding core platform capabilities.
* Balance centralized platform governance with the ability of individual teams to develop specialized products.
* Ensure the architecture integrates effectively with the organization's existing Azure, Snowflake, Databricks and enterprise application ecosystem.

*Architect Agentic AI Products*

* Define reference architectures for AI agents that reason across multistep workflows, use enterprise tools and data, and take actions within controlled boundaries.
* Guide decisions involving deterministic workflows, LLM-driven orchestration, single-agent systems and multi-agent patterns.
* Establish technical patterns for tool calling, structured outputs, context management, memory, retrieval and human approval.
* Design for long-running and asynchronous workflows, state management, failure recovery, retries and idempotency.
* Determine appropriate levels of autonomy based on business value, reliability, security and risk.
* Provide architectural leadership from initial discovery through production deployment.
* Build Shared Platform Capabilities
* Design or guide the implementation of model gateways, approved-model catalogs and provider abstractions.
* Establish reusable RAG, enterprise-search, embedding, vector-search and knowledge-access services.
* Define tool and connector frameworks for securely integrating agents with APIs, databases, SaaS platforms and internal applications.
* Establish agent runtime, orchestration, prompt management and model-versioning capabilities.
* Build evaluation, observability, tracing, feedback and production-monitoring services into the platform.

*Evaluate Platforms and Technology Partners*

* Assess commercial, low-code, open-source and cloud-native AI platforms against the organization's long-term requirements.
* Provide independent architectural oversight for current and future agentic AI pilots.
* Determine which capabilities should be purchased, configured or developed internally.
* Evaluate platform fit across security, integration, scalability, observability, developer experience, cost and governance.
* Identify vendor-lock-in risks and maintain practical portability for critical business logic, data, prompts, evaluations and integrations.
* Define how selected platforms should fit within the broader enterprise architecture rather than becoming isolated technology silos.
* Create a clear path from vendor-supported pilots to sustainable internal ownership.
* Establish Security and Governance
* Design agent and service identity patterns using least-privilege access.
* Define authentication, authorization and user-identity propagation across models, tools, data and applications.

*Operationalize Evaluation and Reliability*

* Define repeatable evaluation frameworks using representative scenarios, golden datasets and automated regression testing.
* Establish measures for task completion, groundedness, tool-selection accuracy, response quality, policy compliance, latency and cost.
* Implement end-to-end logging and tracing that allows teams to understand why an agent made a decision or failed.
* Define production-monitoring, feedback and quality-improvement processes.
* Establish fallback, timeout, retry, escalation and graceful-degradation patterns.

*Drive Delivery and Technical Adoption*

* Partner with the Agentic AI Product Manager to translate business priorities into feasible product and platform roadmaps.
* Work closely with Forward-Deployed AI Engineers, data scientists, software engineers and data engineers during implementation.
* Provide hands-on support for the first production use cases, including prototyping, architecture validation, code reviews and troubleshooting.
* Challenge unnecessary complexity and ensure that agentic AI is applied only where it improves the outcome.

*Mentor and Build Internal Capability*

* Coach experienced engineers, architects and data scientists who are developing agentic AI expertise.
* Lead architecture reviews, code reviews, technical workshops and working sessions.
* Teach practical patterns for agent design, retrieval, evaluation, security, observability and production operations.
* Establish technical standards without becoming a bottleneck to delivery.
* Document reference architectures, reusable patterns, architectural decisions and lessons learned.
* Transfer platform and product ownership to internal teams over time.

**Required Qualifications**

* 10 years of experience across software architecture, cloud platforms, data platforms, machine learning or enterprise application development.
* Significant experience designing and delivering enterprise AI, machine-learning or data platforms.
* Demonstrated experience taking generative AI or agentic AI systems from prototype into production.
* Direct architectural experience with systems involving multistep reasoning, tool use, orchestration or autonomous workflow execution.
* Strong understanding of LLMs, model selection, prompting, structured outputs, tool calling, RAG, embeddings and vector search.
* Experience designing secure integrations with enterprise data, APIs, SaaS platforms and operational systems.
* Strong knowledge of cloud-native architecture, distributed systems, APIs, containers and event-driven or asynchronous processing.
* Experience establishing evaluation, observability, monitoring and operational controls for nondeterministic AI systems.
* Strong understanding of identity, access management, secrets management, data security and least-privilege design.
* Experience making pragmatic build-versus-buy decisions and evaluating enterprise technology vendors.
* Ability to develop reference implementations, review production code and troubleshoot complex technical issues.
* Demonstrated experience mentoring senior engineers, architects or data scientists.
* Strong executive and stakeholder communication skills, including the ability to explain complex architectural decisions in business terms.

**Preferred Qualifications**

* Experience with Azure OpenAI, Azure AI services or comparable managed AI platforms.
* Experience integrating AI applications with Snowflake, Databricks or similar enterprise data environments.
* Experience designing shared model gateways, agent runtimes, tool registries or enterprise AI-development platforms.
* Familiarity with agent frameworks such as Semantic Kernel, LangGraph, OpenAI Agents SDK, AutoGen, CrewAI or comparable technologies.
* Experience implementing Model Context Protocol servers, clients or similar tool-integration standards.
* Familiarity with commercial or low-code agentic AI platforms.
* Experience supporting multiple model providers and balancing portability against unnecessary abstraction.
* Experience with LLMOps, MLOps, prompt and model versioning, automated evaluations and AI governance.
* Experience within retail, apparel, consumer products, merchandising, planning, supply chain, marketing or customer experience.
* Background in consulting, forward-deployed engineering or another environment requiring direct collaboration with business stakeholders.

**Ideal Candidate**

* The ideal candidate combines the enterprise perspective of a platform architect with the practical depth of an applied AI engineer.
* You can evaluate an emerging platform, design a scalable technical foundation and then work directly with engineers to prove that the architecture functions in production.
* You understand that a successful pilot does not automatically constitute an enterprise platform, and that an elegant architecture has little value if teams cannot use it to deliver meaningful products.
* You are thoughtful about where autonomy creates value, skeptical of unnecessary complexity and comfortable making decisions in a rapidly evolving technology landscape.
* Most importantly, you build platforms that accelerate delivery, protect the enterprise and leave internal teams more capable than when you arrived.

**About BICP**

BICP is a San Diego-based AI, Data & Analytics consultancy that helps organizations close the gap between strategic priorities and the specialized capabilities required to execute them. Our Forward Deployed practitioners work alongside client teams, bringing deep expertise across AI/ML, Data, Analytics, Product, and Architecture to move initiatives from idea to implementation, production, and measurable business impact.