CPU-AI Inference Cloud Deployment

Recently I had to design and implement an AI workload cloud architecture. The goal was to deploy 2-3 models that work back to back. Below are the details on the models (The model choices and AI work in this project are done by Abdelrahman Elamrawy). They are all zero shot models, meaning that the models only understand and train on what is currently in the text prompt (pre-inference). The example below in DINO further explains this.

The first model is Grounding DINO. It takes text input and finds that input in the image you feed it. It then draws bounding boxes around the object. For example, if the text prompts are "dog. cat. hamster.", Grounding DINO will try to find these animals in the image you feed it.
The second model named convnext. It gets fed each of the boxes that Grounding DINO has drawn. For each box, it classifies some properties of the animals we're looking at, their breed for example.
The final model is clip, by open AI. It determines the color and pattern of the animals.

Here are the deployment requirements:

Fast. This means < 5 seconds from the user submitting their post request with an image and getting a response.
Cost effective and cheap. This workload is intended for a startup, we cannot afford provisioning a large server with big GPUs running 24/7 to save cold startup times in case users come at any time. For reference, c9g.12xlarge costs 2.08 USD hourly and can handle around 10 requests simultaneously in our case. However, this costs around 1500 USD per month! Even worse, we pay this regardless of how much traffic we have (yes, 1500 USD per month even if you have zero traffic). Ideally, the solution should not cost us more than 100 USD a month when we are facing idle traffic.

Initial exploration

The AI developer gave me a python script which does the inference. Thus, I decided to first write a docker image which loads the models and the sample script I was given and tried to optimize its size. I did this because having a docker image at hand can give us some idea of what options we have in deployment.

The first thing that came to mind was, should I embed the models inside the docker image or make the container pull it at runtime from a cloud storage service like S3? The latter solution messes up our first requirement, as pulling ~ 1.5 gigabytes from S3 would take precious time from our 5 second budget. As such, embedding the models in the docker image made more sense. After optimizing the docker image and using a multi-stage build process, I got the size down to 2.4 GB, which does not affect our cold startup time much compared to starting a 700 mb container then pulling in 1.5 GB of model weights.

            FROM python:3.14-slim AS build
            EXPOSE 80
            WORKDIR /code
            
            RUN python -m venv /opt/venv
            ENV PATH="/opt/venv/bin:$PATH"
            
            COPY ./requirements.txt /code/
            
            # Install PyTorch CPU and your requirements, then immediately hot-swap OpenCV to save space
            RUN pip install --no-cache-dir torch torchvision --index-url https://download.pytorch.org/whl/cpu \
                && pip install --no-cache-dir -r /code/requirements.txt \
                && pip uninstall -y opencv-python \
                && pip install --no-cache-dir opencv-python-headless \
                && rm -rf /root/.cache/pip
            
            COPY ./convnext_animals_traced.pt ./models ./
            COPY ./main.py ./
            RUN python -c "import main"
            
            
            FROM python:3.14-slim AS prod
            EXPOSE 80
            WORKDIR /code
            
            COPY --from=build /opt/venv /opt/venv
            COPY --from=build /code /code
            
            ENV PATH="/opt/venv/bin:$PATH"
            
            CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "80"]
        

I then explored the costs of deploying this in multiple ways.

Note that upon testing this, we concluded that we do NOT need GPUs to meet our performance requirement of < 5 seconds end-to-end. As such, I only designed solutions that use CPU compute.

To keep everything below objective, we assume that we need 8 GB of memory and around 6 vCPUs to run one request with the performance we aim for.

ECS managed CPU instances

I'll have to handle scaling and load balancing myself if I use this for deployment, as ECS does not handle that and only gives you compute.

The managing fee + base cost of c8g.xlarge (8 GB ram and 4 vCPUs) is:

(0.01914+0.15896) * 24 * 30 = 128.2 USD

This is not bad, but remember that this is for only one instance, and that we will definitely need to scale up if we get more than 1 request per 5-6 seconds, so this is the MINIMUM cost.

SageMaker

Amazon SageMaker has a lot of different services and it can get confusing, so let's go through some of them:

Amazon SageMaker Serverless Inference is a purpose-built inference option that enables you to deploy and scale ML models without configuring or managing any of the underlying infrastructure. Sadly though, it has a long boot up time which consists of downloading our docker image and the startup time of the container. While the startup time of the container isn't bad, after starting up the container loads the model weights from its filesystem, and that alone takes 3 seconds. Below is part of our inference script to make this clearer.

                    def load_clip():
                        try:
                            m, _, p = open_clip.create_model_and_transforms("hf-hub:Marqo/marqo-fashionSigLIP")
                            t = open_clip.get_tokenizer("hf-hub:Marqo/marqo-fashionSigLIP")
                        except Exception:
                            m, _, p = open_clip.create_model_and_transforms("ViT-B-32", pretrained="openai")
                            t = open_clip.get_tokenizer("ViT-B-32")
                        m = m.to(device).eval()
                        return m, p, t
        

Amazon SageMaker real time inference does the job but with an INSANE price difference relative to ECS managed instances. We're not charged only at the time of inference here, so this isn't a pure serverless solution and it does not matter if we're getting high or low requests, we pay for what we provision, at a premium cost as well.

SageMaker real-time inference pricing screenshot

If we provision an ml.c8g.xlarge server which has 8 GB ram and 4 vCPUs, we pay 157 USD per month for only running one instance. Using this solution, I will have to handle scaling by telling SageMaker when to scale. For example, scaling to hit a target of 12 requests per minute per instance. This is so far not that bad. Note that we have omitted the volume of data processed from the calculation.

I still do not like needing to handle the admin overhead of scaling though. Why? Because I'll have to predict the usage of this workload I'm deploying and that's just more work for me.

ECS Fargate (CPU only)

Using this, we will have to keep at least 1 task ready for the entire month so that we have no cold-startup time. Since we need a minimum of 8 vCPUs and 10 GB ram, the pricing is as follows:

price per hour = 0.04656 * 6 + 0.00511 * 8 = 0.32024 usd per hour

Since we keep 1 task ready for the entire month, this costs: 231 USD monthly as the starting point. The cost of course increases depending on traffic because we would need more tasks when our user usage increases.

Lambda

This is the fun and interesting one. Using this solution, we can also meet both our performance and cost effectiveness requirements. We can also NOT be responsible for scaling/load balancing, etc.

To meet our performance constraints of having an end to end workload though we can not only use Lambda without its additional features. This is because we would have to load the models when the lambda function is invoked, which again can take up to 3 seconds and that's valuable time. To fix this, we use Lambda snap-start. This introduces multiple constraints though.

Lambda + snap-start first initializes our function and stops right before executing the lambda handler (the main purpose of the lambda function), and then takes a snapshot of the state of the initialized memory and disk state using a "Firecracker" microVM. Afterwards, when a lambda function is invoked, it immediately restores this snapshot and executes the lambda handler after. Restoring the microVM snapshot takes time though (1-2 seconds for our case) , and the heavier the snapshot is (e.g. due to big models loaded into the lambda function's memory), the longer this restore time will take. The 1-2 seconds restore time is worth it from a trade-off point of view because we get incredibly easy scaling and cost effectiveness as a startup business when using Lambda. Moreover, when multiple invocations are happening there is always the benefit of warm starts so this restore time is reduced to 0 seconds.

Note that, although the restore time of the snapshot of Lambda after its initialization phase is only dependent on the snapshot's size, AWS bills you for the lambda function's entire memory as if it was used as the snapshot size. This will be apparent when we are calculating the cost below.

The hurdle we have to overcome is that Lambda + snap-start limits our /tmp storage to 512 mb. While this seems fine, we use pytorch for inference. The pytorch CPU wheel on its own is like 700 megabytes (iirc) and so that's a problem. We also need to fit all our dependencies in a lambda layer which is only 250 mb, unzipped. Yikes. The approach mentioned here gives us the first step in what we need to do though.

Basically, we zip the heavy dependencies inside the lambda layer while keeping other dependencies unzipped. When the lambda function is invoked, we unzip these dependencies in /tmp and use them as needed. If you do this with the CPU wheel of pytorch, you will unfortunately fail because it's still too large. The solution to this is to build torch from source with ONLY the things we will use in our inference. This of course depends on what your models do and can be a trial and error process. Below are the other dependencies in our lambda layer.

            pillow
            python-dotenv
            requests
            transformers==4.42.4
            safetensors
            sympy
            numpy<2.0,>=1.23
        

I built the lambda layer using a docker to make use of layer caching. This is beneficial because torch takes around 1 hour to build. Using layer caching, I need to only build once (with ENV flags unchanged) and then can change any step after that with no problem. The Dockerfile for building the lambda layer is below. Note that we're building torchvision so its compiled extensions link directly against our built torch.

            # IMPORTANT: base image changed from python:3.12-slim (Debian) to the
            # ACTUAL Lambda runtime image. This is the fix for the glibc mismatch we
            # hit during local testing: python:3.12-slim ships a newer glibc than the
            # real Lambda execution environment.
            FROM public.ecr.aws/lambda/python:3.12 AS builder
            
            # libgomp-devel pulls in the matching headers/symlinks for OpenMP,
            # so the wheel links against THIS image's
            # libgomp rather than needing to source it separately afterward.
            # libjpeg-turbo-devel/libpng-devel are added here for torchvision's image
            # codec extensions (decode_jpeg/decode_png)
            RUN dnf install -y \
                gcc \
                gcc-c++ \
                make \
                cmake \
                git \
                libgomp \
                libjpeg-turbo-devel \
                openblas-devel \
                libpng-devel \
                && dnf clean all \
                && rm -rf /var/cache/dnf
            
            RUN pip3 install --no-cache-dir --upgrade pip setuptools wheel pyyaml typing_extensions
            RUN pip3 install --no-cache-dir "numpy<2.0,>=1.23"
            
            WORKDIR /tmp
            
            RUN git clone --depth 1 --branch v2.3.1 --recurse-submodules --shallow-submodules https://github.com/pytorch/pytorch.git
            
            WORKDIR /tmp/pytorch
            
            # The flags we need for our inference to work. Change based on your inference script.
            ENV BUILD_TEST=0 \
                USE_DISTRIBUTED=0 \
                USE_NCCL=0 \
                USE_GLOO=0 \
                USE_TENSORPIPE=0 \
                USE_MKL=0 \
                USE_MKLDNN=1 \
                BLAS=OpenBLAS \
                USE_CUDA=0 \
                USE_ROCM=0 \
                USE_GFLAGS=0 \
                USE_GLOG=0 \
                USE_NNPACK=0 \
                USE_QNNPACK=0 \
                USE_PYTORCH_QNNPACK=0 \
                USE_XNNPACK=0 \
                USE_KINETO=0 \
                USE_FBGEMM=0 \
                USE_BREAKPAD=0 \
                USE_ITT=0 \
                CMAKE_BUILD_TYPE=Release \
                MAX_JOBS=2 \
                CFLAGS="-O2" \
                CXXFLAGS="-O2"
            RUN python3 setup.py clean && python3 setup.py bdist_wheel
            
            # We have to build torchvision against THIS torch, not the official PyPI one
            # Reason: torchvision's compiled extension (torchvision._C) links directly
            # against libtorch's C++ symbols. The official PyPI torchvision wheel is
            # built against the official PyPI torch wheel's exact ABI settings (e.g.
            # -D_GLIBCXX_USE_CXX11_ABI). Building from source here, in the same
            # environment right after installing the torch wheel we just built,
            # guarantees the ABI matches.
            #
            # v0.18.1 is the torchvision release matched to torch v2.3.1
            # (see https://github.com/pytorch/vision#installation compatibility table).
            #
            # FORCE_CUDA=0 / the TORCHVISION_USE_* flags disable GPU codec paths
            # (nvjpeg, ffmpeg video decoding) that aren't needed for CPU image
            # classification and would otherwise pull in extra native deps.
            RUN pip3 install --no-cache-dir dist/torch-*.whl
            
            WORKDIR /tmp
            RUN git clone --depth 1 --branch v0.18.1 --recurse-submodules --shallow-submodules https://github.com/pytorch/vision.git
            
            
            WORKDIR /tmp/vision
            RUN pip3 install --no-cache-dir "setuptools<82"
            ENV FORCE_CUDA=0 \
                TORCHVISION_USE_NVJPEG=0 \
                TORCHVISION_USE_VIDEO_CODEC=0 \
                TORCHVISION_USE_FFMPEG=0 \
                MAX_JOBS=2 \
                CFLAGS="-O2" \
                CXXFLAGS="-O2"
            RUN python3 setup.py bdist_wheel
            
            
            # STAGE 2: Build the Lambda Layer Zip Artifact
            FROM public.ecr.aws/lambda/python:3.12 AS layer-generator
            
            # binutils (for `strip`) and zip -- AL2023 package names.
            RUN dnf install -y \
                binutils \
                zip \
                libgomp \
                findutils \
                openblas-devel \
                libgfortran \
                && dnf clean all \
                && rm -rf /var/cache/dnf
            
            WORKDIR /workspace
            RUN mkdir -p /workspace/python
            
            # Install torch+torchvision together, with --no-deps, from Stage 1 build
            # --no-deps is critical here: without it, pip would see torchvision's
            # declared "torch==2.3.1" requirement and, since --target is just a plain
            # directory (not a resolver-visible environment), go fetch the OFFICIAL
            # torch wheel from PyPI to "satisfy" it
            COPY --from=builder /tmp/pytorch/dist/torch-*.whl .
            COPY --from=builder /tmp/vision/dist/torchvision-*.whl .
            RUN pip install --no-deps --no-cache-dir --target /workspace/python/ \
                    torch-*.whl torchvision-*.whl \
                && rm torch-*.whl torchvision-*.whl
            
            # Install remaining runtime dependencies from requirements.txt
            COPY req_layer.txt .
            RUN pip install --no-cache-dir --target /workspace/python/ -r req_layer.txt
            
            # This next step is necessary because the Lambda runtime doesn't include
            # the dnf installs we made above (libgomp, libopenblas, libgfortran).
            # 1. Grab the main OpenBLAS library (using find so we don't assume the exact symlink name)
            RUN cp -L $(find /usr/lib64 -name "libopenblas.so.*" | head -n 1) /workspace/python/torch/lib/libopenblas.so.0
            
            # 2. Use ldd to find EVERY dynamic dependency of every .so in torch/lib.
            # 3. Filter out the core system libraries that Lambda already provides natively
            #    (glibc, libm, libgcc, libstdc++, etc.).
            # 4. Copy the remaining required libraries (Fortran, OpenMP, QuadMath, etc.)
            #    directly into torch/lib so they get zipped up.
            RUN find /workspace/python/torch/lib/ -type f -name "*.so*" -exec ldd {} \; 2>/dev/null \
                | grep "=> /" \
                | awk '{print $3}' \
                | sort -u \
                | grep -Ev '/libc\.so|/libm\.so|/libpthread\.so|/libdl\.so|/librt\.so|/libresolv\.so|/libgcc_s\.so|/libstdc\+\+\.so|/ld-linux' \
                | xargs -I {} cp -n -L {} /workspace/python/torch/lib/ \
                && ls -la /workspace/python/torch/lib/
            
            
            # These are unnecessary
            RUN rm -rf /workspace/python/torch/include
            RUN rm -rf /workspace/python/torch/share
            
            # torch/bin: keep this directory, but only keep torch_shm_manager inside
            # because removing it crashes torch on import
            RUN find /workspace/python/torch/bin -type f ! -name 'torch_shm_manager' -delete 2>/dev/null || true
            
            # Squeeze out remaining binary symbol table bloat from shared objects.
            # --strip-unneeded preserves the dynamic symbol table (.dynsym) needed for
            # Python's C extension loader / pybind11 boundary, while removing debug symbols
            RUN find /workspace/python/ -name "*.so" -exec strip --strip-unneeded {} \;
            
            
            # More pruning
            RUN rm -rf /workspace/python/hf_xet* \
                && find /workspace/python/numpy -type d -name "tests" -exec rm -rf {} + \
                && rm -rf /workspace/python/numpy/f2py \
                && rm -rf /workspace/python/numpy/distutils \
                && rm -rf /workspace/python/numpy/doc
            
            # Zip heavy libraries separately INSIDE python/
            RUN cd /workspace \
                && zip -r9 python/torch.zip python/torch python/torch-*.dist-info \
                && zip -r9 python/sympy.zip python/sympy python/sympy-*.dist-info \
                && zip -r9 python/transformers.zip python/transformers python/transformers-*.dist-info \
                && rm -rf /workspace/python/torch /workspace/python/sympy /workspace/python/transformers
            
            # Now python/ contains the .zip files, so they will be included here:
            RUN cd /workspace && zip -r9 /workspace/pytorch-lambda-layer.zip python/
            CMD ["/bin/bash"]
        

Of course, you always have to test whether unzipping the heavy libraries onto a separate directory is < 512 mb, as well as if the unzipped layer is < 250 mb so we can use Lambda snap-start.

Notice that up until now we have not mentioned where our models are. 1.5 gigabytes will obviously not fit in our /tmp directory. A neat solution for this is loading our models from S3 directly into memory in our initialization phase though! (source) We do that using memfd, which you can read more about in the AWS article I linked. This eats up some of our memory, so make sure you allocate an extra 1.5 gigs to get the same performance you aim for.

Funnily enough, when I was doing the pricing calculation for this using the AWS Pricing calculator, I found a bug and reported it on AWS re:Post. You can check my post out here.

The calculation is as follows: Lambda gives you 6 vCPUs when you allocate 10 GB of memory, so let's do that so we get a fair comparison to the other services above. Lambda + snap-start charges us for the following:

Number of requests
Snap-start restores: number of snap-start cold-starts X amount of memory allocated
Lambda compute used: duration of each request X amount of memory allocated
Snap-start cache: basically the amount of time our lambda function has snap-start enabled, this is measured in GB-seconds and can be calculated by: time of which snap-start is enabled X amount of memory allocated

We use 3 scenarios to compare this solution to others: (1) Zero traffic per month, (2) regular medium traffic per month (17k requests per day), and (3) bursty traffic for 10 days a month, each with 25k requests per day.

Scenario 1: zero traffic per month

The charges are, in the order mentioned above respectively:

Number of requests: 0 USD
Snap-start restores: 0 USD
Lambda compute used: 0 USD
Snap-start cache: If we use 10 GB memory and we have snap-start enabled for the entire month: 0.0000015046 per GB-second * 10 GB * 30 * 24 * 60 * 60 = 39 USD

So a total of 39 USD per month if we idle. Compared to all the other solutions we have, this is a clear winner.

Scenario 2: Regular predictable traffic per month

Why 17k requests? Because the other solutions could theoretically be used 24/7 for back to back requests, and if every request is 5 seconds, we could fit 17.2 k requests per day. Of course, in online applications this never happens and instead we see sporadic requests throughout the day with some hours having a lot more traffic than others, which the other solutions above fail to fulfill unless they scale at these busy times, coming at an extra cost.

The charges are, in the order mentioned above respectively:

Number of requests: 0.2 USD per 1 Million request, so if 17k every day for a month we get: 0.2/(10^6) * (17 * (10^3) * 30) = 0.102 USD
Snap-start restores: We only get 1 cold start at the start of the month and then every invocation after that uses the same warm lambda function since they're back to back 24/7, but supposing that is not the case and that we do cold-starts for 10% of invocations: 0.1 * 0.0001397998 for every GB restored * 10 GB * 17 * (10^3) * 30 = 72 USD
Lambda compute used: 0.0000166667 for every GB-second * 5 seconds * 10 GB * 17 * (10^3) * 30 = 425 USD
Snap-start cache: 39 USD - This does not vary with traffic and we keep it enabled for the entire month

We get a total of 0.102 + 72 + 425 + 39 = 526.1 USD. Although this scenario is very unlikely, we have to admit in this case this solution is worse cost wise than the others.

Scenario 3: Bursty traffic for 10 days a month, each with 25k requests per day

Number of requests: 0.2 USD per 1 Million request, so if 25k every day for a month we get: 0.2/(10^6) * (25 * (10^3) * 10) = 0.05 USD
Snap-start restores: Let's assume this time we get 25% cold-starts because of the bursty nature of this scenario, and that the traffic comes while everything is cold: 0.25 * 0.0001397998 for every GB restored * 10 GB * 25 * (10^3) * 10 = 87.4 USD
Lambda compute used: 0.0000166667 for every GB-second * 5 seconds * 10 GB * 25 * (10^3) * 10 = 208.3 USD
Snap-start cache: 39 USD

We get a total of 0.05 + 87.4 + 208.3 + 39 = 334.8 USD. This is a really good cost for such a scenario, and not only that, we do not handle scaling ourselves using auto scaling groups or scaling policies like in SageMaker! Moreover, other solutions wouldn't scale as fast as lambda and would have a very long cold startup time (30 seconds to 1 minute as a minimum) when scaling, compared to this almost instant startup time: 1.5 second snap-start microVM restore.

This solution is by far the winner. I won't go into the details of setting up an API gateway to invoke this lambda function because that's trivial, you could check out this if necessary.

Conclusions and takeaways

In this project I have shown that:

You can run CPU AI inference cost effectively on the cloud by utilizing AWS Lambda + snap-start. This can be done by fitting pytorch and more heavy dependencies in a lambda layer by building torch from source and zipping it + other dependencies then unzipping to /tmp storage at runtime.
Lambda snap-start allows you to have very small to negligible cold-startup times.
To overcome /tmp storage 512 mb limit for snap-start you could load model weights from S3 directly into the lambda function's memory in the initialization phase. Although this makes the initialization phase take more time, Lambda snap-start makes this irrelevant since we take a snapshot of the memory and disk state of the lambda function using Firecracker microVM snapshots.

I have also:

Compared and contrasted performance and cost effectiveness of different deployment strategies for an AI workload using: ECS managed instances, ECS fargate, SageMaker, and Lambda.
Discovered and reported an AWS pricing calculator bug for Lambda snap-start cache. Details here.