Serverless functions are great for lightweight cloud architecture and rapid provisioning. However, sometimes serverless introduces additional complexity to the deployment process. Author compares Python and Go with respect to the ease of deployment when setting up a simple data factory on AWS Lambda. Written by Dick Abma.
Serverless functions are a great way to simplify your cloud stack and quickly produce new functionality. Serverless contributes to scalability, transferability, and flexibility of the entire stack. For author as a data engineer, this means that he can produce applications that will be understood by clients and can adapt to evolving requirements.
The article describes:
- Case study - bunch of IoT devices add data to a database, which is accessible via a REST API
- CI/CD: Platform compatibility
- Python in AWS Lambda
- Golang on AWS Lambda
- AWS limits, concurrent requests and validation
- Conclusion deserialization
The article covers more realistic setup with a Lambda Layer, in which you can place the needed packages. The layer is loaded by the function when it is invoked. Packages are thus removed from the function itself.
Serverless functions like AWS Lambda are great for lightweight cloud architecture. Though we should be careful not to introduce complexity in our CI/CD and Terraform code, which may outweigh the advantages of Lambda. Code is provided in this GitHub repository. Great read!
[Read More]