Deployment is a critical phase in the software development lifecycle that involves releasing a solution into a live environment where it can be accessed and used by end-users. This article explores the significance of deployment and provides insights into effective deployment strategies, considerations, and best practices. From selecting the right deployment approach to ensuring scalability and monitoring, we delve into the key aspects of successful deployment, empowering organizations to seamlessly deliver their solutions to production.

Deployment for inference

• FasterTransformer and Triton server

  • Highly optimized layer between the transformer and the GPUs
  • Multiple model instances, batching, and other features

• Auto-scaling infrastructure using k8s

  • Unique challenges like model size and GPU requirements
  • Dealing with GPU shortages in individual zones