Requirement / include for package from local path

[ronyis]

I have a project in which multiple services are defined on a single repository, with many dependencies between them (we have considered splitting the repo but we prefer it like that in order to have a single version tag, and easier development process).

The repository looks like:

service1/
---serverless.yml
---requirements.txt
---handler1.py
---service1/
------__init__.py
------resource1.py
service2/
---serverless.yml
---requirements.txt
---handler2.py
---service2/
------__init__.py
------resource2.py
utils/
---__init__.py
---general.py

Each service should be able to import modules from any other service.
Since it's not possible on Serverless framework to add files from an outer directory, I was thinking to use the 'vendor' option for doing that (as offered in #168). The problem that it allows me to choose only a single directory. Using an upper directory isn't possible due to recursion error.

Another (preferred) approach is to install each package with pip, but using local paths in the requirements file doesn't work since they are not available in the docker.

The only solution for me now is to use symbolic links which is much less convenient (and still doesn't let me use local paths in the requirements file for real package installation).

Any solution for that?

[AndrewFarley]

First, with what you've describe above if all the folders symlink to other folders it might do so recursively and would cause you problems of infinite depth. I'd be cautious of this (untested, maybe links aren't followed with infinite depth).

@ronyis I must raise a serious question about your folder hierarchy and your usage pattern. What you have described looks like you are using packaging individually, hence the different requirements.txt files. However... if you package individually, each function will only have the requirements from that function's requirements.txt file. So the code from other functions if they use a requirement that isn't in your function, won't work. So... this... raises my first recommendation...

First, one solution might be not packaging individually, I do this for one of my clients, it actually speeds up deploys a LOT in both the packaging stage and the deployment phase as well. To do this you'd use a different handler path for each function. Eg: handler: service2/handler2.lambda_handler and handler: service1/handler1.lambda_handler.

And then, as you mentioned another solution is indeed using vendor, which you can do differently per function declaration, so you can include another function as a "vendor". Unfortunately it is currently limited to "one" vendor folder. So you could technically set the "root" path as the vendor, basically including everything. Or you could include a "single" other function, but you couldn't (currently) pick and choose what others to include. And with that bug you linked, include doesn't work properly so you may have to use vendor as a pseudo-include mechanism. Hope that helps?

[ronyis]

Thanks for help!

Few things:

• There shouldn't be a case where two packages use each other, so a recursive error shouldn't happen.

• Each of the services has a different serverless.yml file, so that I don't have to use the individual packaging feature. This is needed because when I work on one of them I don't want to deploy all the system (consisting of Lambdas and resources). Nevertheless, the problem with the requirements you mentioned could happen - and therefore I will be glad to find a solution that will install each package with pip. Anyway, I think that even if I just import the code I need from other services and add the specific requirements needed again on my service won't be too bad (but it's definitely worse).

• If I use the "root" directory as "vendor", a recursive error happen because it is tried to be copied into a subdirectory.

• About the bug I've linked - I'm not trying to use 'include', since it not able to add filed from an outer directory of which 'serverless.yml' is defined. I just referenced it as an example of 'vendor' feature usage to copy files to your artifact.

[AndrewFarley]

Ahhh, I just noticed, these are all complete individual stacks! Yeah, symlinks are probably your best bet then, if you just avoid circular linking logic, I think you'd be hard pressed to find an alternate to this. I guess to work around the potential missing package thing you could copy any symlinked'd requirements contents into the current function's requirements.txt file. This sounds like hell if you have a lot of cross-function dependency... but if you have good comments in your requirements.txt you can categorize the items in there to make it easier to keep track of. And you're right, you can't use vendor outside of the folder. So symlinks is probably your only option.

[ronyis]

So do you mean to manually copy the requirements, right?

About the vendor possibility - I'm considering to extract the "common" code from each package and put it all in the same folder (so that I will have a single folder to include). What do you think about that?

[AndrewFarley]

@ronyis The common code thing is indeed the best idea... but since it will live in the root directory you'd still need to symlink to that folder into each and every project's folder since it will be above their folder. Thankfully you can commit (in git) symlinks, so this should work everywhere. Vendor isn't going to be useful to you, since symlinks already do what you want here.

And yes, about requirements I mean if func 1 is going to symlink func 2, you copy the contents of func2's requirements into func1's requirements.

[ronyis]

I have ended up using another plugin - serverless-package-common which creates the symlinks temporarily for me.

I think that a useful feature for this plugin will be to enable installation of a package from a local directory (probably by copying it into the docker).

@AndrewFarley Thanks again for the advice.

[miketheman]

Glad to hear you sorted out the behavior with the other plugin, sorry this didn’t work out. M going to close this aged issue, and we can revisit this if the use case arises again.