Serverless - Packaging Dependencies

In the previous chapter, we saw how to use plugins with serverless. We specifically looked at the Python Requirements plugin and saw how it can be used to bundle dependencies like numpy, scipy, pandas, etc.with your lambda function's application code. We even saw an example of deploying a function requiring the numpy dependency. We saw that it ran well locally but on the AWS Lambda console, you'd have encountered an error if you are on a Windows or Mac machine. Let's understand why the function runs locally but doesn't run after deployment.

If you look at the error message, you get some hints. I'm specifically referring to one line − 'Importing the numpy C-extensions failed.' Now, many important python packages like numpy, pandas, scipy, etc.require the compilation of C-extensions. If we compile them on a Windows or a Mac machine, then Lambda (linux environment) will throw up an error when trying to load them. So the important question is, what can be done to avoid this error. Come in docker!

What is docker?

According to Wikipedia, docker is a set of platform as a service (PaaS) products that use OS-level virtualization to deliver software in packages called containers. If you scan the Wikipedia page of docker a bit more, you will come across some more relevant statements − Docker can package an application and its dependencies in a virtual container that can run on any Linux, Windows, or macOS computer. This enables the application to run in a variety of locations, such as on-premises, in a public cloud, and/or in a private cloud. I think it should be very clear after the above statements. We have an error coming up because C-extensions compiled on Windows/Mac don't work in Linux.

We can simply bypass that error by packaging the application in a container that can run on any OS. What docker does in the background to achieve this OS-level virtualization is beyond the scope of this chapter.

Installing docker

You can head over to https://docs.docker.com/engine/install/ for the installation of Docker Desktop. If you are using Windows 10 Home Edition,the Windows version should be at least 1903 (May 2019 update). Therefore, you may want to upgrade your Windows 10 OS before installing Docker Desktop. No such limitations apply to Windows Pro or Enterprise versions.

Using dockerizePip in serverless

Once Docker Desktop has been installed on your machine, you need to make only the following addition to your serverless.yml file to package your applications and dependencies using docker −

custom:
   pythonRequirements:
      dockerizePip: true

Please note, that if you have been following along with me since the previous chapter, it is likely that you have already deployed code to lambda once.This would have created a static cache in your local storage. By default, serverless would use that cache to bundle dependencies, and therefore, docker container won't be created.Therefore, to force serverless to create use docker, we will add another statement to pythonRequirements −

custom:
   pythonRequirements:
      dockerizePip: true
	useStaticCache: false #not necessary if you will be deploying the code to lambda for the first time.

This last statement is not necessary if you are deploying to lambda for the first time. In general, you should set useStaticCache to true, since that will save you some packaging time when you haven't made any changes to the dependencies or the way they have to be packaged.

With these additions, the serverless.yml file now looks like −

service: hello-world-python
provider:
   name: aws
   runtime: python3.6
   profile: yash-sanghvi
   region: ap-south-1

functions:
   hello_world:
      handler: handler.hello
      timeout: 6
      memorySize: 128
	
plugins:
   - serverless-python-requirements
  
custom:
   pythonRequirements:
      dockerizePip: true
	useStaticCache: false #not necessary if you will be deploying the code to lambda for the first time.

Now, when you run the sls deploy -v command, make sure that docker is running in the background. On Windows, you can just search for Docker Desktop in the Start menu and double click on the app. You will soon get a message that it is running. You can also verify this through the small popup near the battery icon in Windows. If you can see the docker icon there, it is running.

Docker

Now when you run your function on the AWS Lambda console, it would work. Congratulations!!

However, in your 'Function Code' section on the AWS Lambda console, you would be seeing a message saying 'The deployment package of your Lambda function "hello-world-python-dev-hello_world" is too large to enable inline code editing. However, you can still invoke your function.'

Function Code

Seems like the addition of the Numpy dependency has made the bundle size too large and as a result, we cannot even edit our application code in the lambda console. How do we solve that problem? Head on to the next chapter to find out.

References

Advertisements