Integration test suite is broken #47

skwashd · 2021-09-04T09:07:43Z

The tests aren't running properly because of duplicated container names. Further subsequent invocations create further failures because the tear down step is missing some of the containers.

Despite these failures being noted in the console, the test suite only catches one of the failures on the first run. On the second run there are more errors, but the suite passes.

First Invocation

$ make integ-tests-with-docker
go test ./...
?   	go.amzn.com/cmd/aws-lambda-rie	[no test files]
ok  	go.amzn.com/lambda/agents	(cached)
ok  	go.amzn.com/lambda/appctx	(cached)
ok  	go.amzn.com/lambda/core	(cached)
ok  	go.amzn.com/lambda/core/directinvoke	(cached)
?   	go.amzn.com/lambda/core/statejson	[no test files]
?   	go.amzn.com/lambda/extensions	[no test files]
?   	go.amzn.com/lambda/fatalerror	[no test files]
?   	go.amzn.com/lambda/interop	[no test files]
ok  	go.amzn.com/lambda/logging	(cached)
ok  	go.amzn.com/lambda/metering	(cached)
ok  	go.amzn.com/lambda/rapi	(cached)
ok  	go.amzn.com/lambda/rapi/handler	(cached)
ok  	go.amzn.com/lambda/rapi/middleware	(cached)
ok  	go.amzn.com/lambda/rapi/model	(cached)
?   	go.amzn.com/lambda/rapi/rendering	[no test files]
ok  	go.amzn.com/lambda/rapid	(cached)
ok  	go.amzn.com/lambda/rapidcore	(cached)
ok  	go.amzn.com/lambda/rapidcore/env	(cached)
?   	go.amzn.com/lambda/rapidcore/standalone	[no test files]
?   	go.amzn.com/lambda/rapidcore/telemetry	[no test files]
?   	go.amzn.com/lambda/rapidcore/telemetry/logsapi	[no test files]
ok  	go.amzn.com/lambda/runtimecmd	(cached)
ok  	go.amzn.com/lambda/telemetry	(cached)
docker run --env GOPROXY=direct -v /path/to/aws-lambda-runtime-interface-emulator:/LambdaRuntimeLocal -w /LambdaRuntimeLocal golang:1.14 make compile-lambda-linux
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -ldflags "-s -w" -o bin/aws-lambda-rie ./cmd/aws-lambda-rie
go: downloading github.com/google/uuid v1.1.2
go: downloading github.com/sirupsen/logrus v1.6.0
go: downloading github.com/jessevdk/go-flags v1.4.0
go: downloading github.com/go-chi/chi v4.1.2+incompatible
go: downloading github.com/go-chi/render v1.0.1
go: downloading golang.org/x/sys v0.0.0-20190422165155-953cdadca894
python3 -m venv .venv
.venv/bin/pip install --upgrade pip
Requirement already satisfied: pip in ./.venv/lib/python3.9/site-packages (21.1.3)
Collecting pip
  Using cached pip-21.2.4-py3-none-any.whl (1.6 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 21.1.3
    Uninstalling pip-21.1.3:
      Successfully uninstalled pip-21.1.3
Successfully installed pip-21.2.4
.venv/bin/pip install requests
Collecting requests
  Using cached requests-2.26.0-py2.py3-none-any.whl (62 kB)
Collecting certifi>=2017.4.17
  Using cached certifi-2021.5.30-py2.py3-none-any.whl (145 kB)
Collecting urllib3<1.27,>=1.21.1
  Using cached urllib3-1.26.6-py2.py3-none-any.whl (138 kB)
Collecting idna<4,>=2.5
  Using cached idna-3.2-py3-none-any.whl (59 kB)
Collecting charset-normalizer~=2.0.0
  Using cached charset_normalizer-2.0.4-py3-none-any.whl (36 kB)
Installing collected packages: urllib3, idna, charset-normalizer, certifi, requests
Successfully installed certifi-2021.5.30 charset-normalizer-2.0.4 idna-3.2 requests-2.26.0 urllib3-1.26.6
.venv/bin/python3 test/integration/local_lambda/end-to-end-test.py
[+] Building 11.1s (8/8) FINISHED
 => [internal] load build definition from Dockerfile-allinone                                                                                                                                                                       0.0s
 => => transferring dockerfile: 217B                                                                                                                                                                                                0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                   0.0s
 => => transferring context: 2B                                                                                                                                                                                                     0.0s
 => [internal] load metadata for public.ecr.aws/lambda/python:3.8                                                                                                                                                                  11.0s
 => [1/3] FROM public.ecr.aws/lambda/python:3.8@sha256:af98ec7625cc27f3efb189a753f57be48d4bedf4f558770157f3651e8bcb56a6                                                                                                             0.0s
 => [internal] load build context                                                                                                                                                                                                   0.0s
 => => transferring context: 1.71kB                                                                                                                                                                                                 0.0s
 => CACHED [2/3] WORKDIR /var/task                                                                                                                                                                                                  0.0s
 => CACHED [3/3] COPY ./ ./                                                                                                                                                                                                         0.0s
 => exporting to image                                                                                                                                                                                                              0.0s
 => => exporting layers                                                                                                                                                                                                             0.0s
 => => writing image sha256:e26aedd62dcec294fd7314bed8482bb9073623c1c2ff2a005eb6ac1bb6740d36                                                                                                                                        0.0s
 => => naming to docker.io/library/aws-lambda-local:testing                                                                                                                                                                         0.0s
0a1ac6973c0b3af7da123e9b4b83c15a94030dd64b51a224ad38a416bd082443
Ee33c68dd37c9e05bf78dc5a9ac4c717dd4dff152d8253c906c7ad39679796654
.635983bf9fe8d2eaef56cb90ed5a1716992fd883da68d62bacffc42e81f534ca
.707bea10bf31192a7b0e7a98fa07fa983607207a5c47461c11a5c66575b10b85
.61c45528518d5d528b8b49bc548de994c212504850f44ffb920f112ce33d4b19
.d8c0d8aed403fbe34af3199903fcd365a0583a4d84cf7f30e254ca13cedf9b97
.docker: Error response from daemon: Conflict. The container name "/testing" is already in use by container "d8c0d8aed403fbe34af3199903fcd365a0583a4d84cf7f30e254ca13cedf9b97". You have to remove (or rename) that container to be able to reuse that name.
See 'docker run --help'.
.b3fc2c0bdd914fc3d29efbc3992c2a213fb5cc08be97d5c5804144b45c4d5fb5
.docker: Error response from daemon: Conflict. The container name "/testing" is already in use by container "d8c0d8aed403fbe34af3199903fcd365a0583a4d84cf7f30e254ca13cedf9b97". You have to remove (or rename) that container to be able to reuse that name.
See 'docker run --help'.
.envvarcheck
testing
timeout
exception
Error response from daemon: conflict: unable to remove repository reference "aws-lambda-local:testing" (must force) - container 635983bf9fe8 is using its referenced image e26aedd62dce
[+] Building 0.4s (8/8) FINISHED
 => [internal] load build definition from Dockerfile-python36                                                                                                                                                                       0.0s
 => => transferring dockerfile: 217B                                                                                                                                                                                                0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                   0.0s
 => => transferring context: 2B                                                                                                                                                                                                     0.0s
 => [internal] load metadata for public.ecr.aws/lambda/python:3.8                                                                                                                                                                   0.3s
 => [internal] load build context                                                                                                                                                                                                   0.0s
 => => transferring context: 107B                                                                                                                                                                                                   0.0s
 => [1/3] FROM public.ecr.aws/lambda/python:3.8@sha256:af98ec7625cc27f3efb189a753f57be48d4bedf4f558770157f3651e8bcb56a6                                                                                                             0.0s
 => CACHED [2/3] WORKDIR /var/task                                                                                                                                                                                                  0.0s
 => CACHED [3/3] COPY ./ ./                                                                                                                                                                                                         0.0s
 => exporting to image                                                                                                                                                                                                              0.0s
 => => exporting layers                                                                                                                                                                                                             0.0s
 => => writing image sha256:e26aedd62dcec294fd7314bed8482bb9073623c1c2ff2a005eb6ac1bb6740d36                                                                                                                                        0.0s
 => => naming to docker.io/library/aws-lambda-local:testing-py36                                                                                                                                                                    0.0s
cb043dfe61f2b6f6292495ec60baf90faeebe0bd55d0c0e7398dd3ceb79dfbfb
.b59248a9811888eec24fa94bf6990cbccddc2d0176337f65b5f591c022e0992a
.testing
assert-overwritten
Untagged: aws-lambda-local:testing-py36

======================================================================
ERROR: test_context_get_remaining_time_in_default_deadline (__main__.TestEndToEnd)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/path/to/aws-lambda-runtime-interface-emulator/.venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/path/to/aws-lambda-runtime-interface-emulator/.venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 445, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/path/to/aws-lambda-runtime-interface-emulator/.venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 440, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/local/Cellar/[email protected]/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1349, in getresponse
    response.begin()
  File "/usr/local/Cellar/[email protected]/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 316, in begin
    version, status, reason = self._read_status()
  File "/usr/local/Cellar/[email protected]/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 285, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/path/to/aws-lambda-runtime-interface-emulator/.venv/lib/python3.9/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/path/to/aws-lambda-runtime-interface-emulator/.venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/path/to/aws-lambda-runtime-interface-emulator/.venv/lib/python3.9/site-packages/urllib3/util/retry.py", line 532, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/path/to/aws-lambda-runtime-interface-emulator/.venv/lib/python3.9/site-packages/urllib3/packages/six.py", line 769, in reraise
    raise value.with_traceback(tb)
  File "/path/to/aws-lambda-runtime-interface-emulator/.venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/path/to/aws-lambda-runtime-interface-emulator/.venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 445, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/path/to/aws-lambda-runtime-interface-emulator/.venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 440, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/local/Cellar/[email protected]/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1349, in getresponse
    response.begin()
  File "/usr/local/Cellar/[email protected]/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 316, in begin
    version, status, reason = self._read_status()
  File "/usr/local/Cellar/[email protected]/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 285, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/path/to/aws-lambda-runtime-interface-emulator/test/integration/local_lambda/end-to-end-test.py", line 142, in test_context_get_remaining_time_in_default_deadline
    r = requests.post("http://localhost:9006/2015-03-31/functions/function/invocations", json={})
  File "/path/to/aws-lambda-runtime-interface-emulator/.venv/lib/python3.9/site-packages/requests/api.py", line 117, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/path/to/aws-lambda-runtime-interface-emulator/.venv/lib/python3.9/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/path/to/aws-lambda-runtime-interface-emulator/.venv/lib/python3.9/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/path/to/aws-lambda-runtime-interface-emulator/.venv/lib/python3.9/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/path/to/aws-lambda-runtime-interface-emulator/.venv/lib/python3.9/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

----------------------------------------------------------------------
Ran 11 tests in 43.669s

FAILED (errors=1)
make: *** [integ-tests-with-docker] Error 1

Second Invocation

$ make integ-tests
python3 -m venv .venv
.venv/bin/pip install --upgrade pip
Requirement already satisfied: pip in ./.venv/lib/python3.9/site-packages (21.2.4)
.venv/bin/pip install requests
Requirement already satisfied: requests in ./.venv/lib/python3.9/site-packages (2.26.0)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in ./.venv/lib/python3.9/site-packages (from requests) (1.26.6)
Requirement already satisfied: charset-normalizer~=2.0.0 in ./.venv/lib/python3.9/site-packages (from requests) (2.0.4)
Requirement already satisfied: certifi>=2017.4.17 in ./.venv/lib/python3.9/site-packages (from requests) (2021.5.30)
Requirement already satisfied: idna<4,>=2.5 in ./.venv/lib/python3.9/site-packages (from requests) (3.2)
.venv/bin/python3 test/integration/local_lambda/end-to-end-test.py
[+] Building 1.1s (8/8) FINISHED
 => [internal] load build definition from Dockerfile-allinone                                                                                                                                                                       0.0s
 => => transferring dockerfile: 217B                                                                                                                                                                                                0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                   0.0s
 => => transferring context: 2B                                                                                                                                                                                                     0.0s
 => [internal] load metadata for public.ecr.aws/lambda/python:3.8                                                                                                                                                                   1.0s
 => [1/3] FROM public.ecr.aws/lambda/python:3.8@sha256:af98ec7625cc27f3efb189a753f57be48d4bedf4f558770157f3651e8bcb56a6                                                                                                             0.0s
 => [internal] load build context                                                                                                                                                                                                   0.0s
 => => transferring context: 107B                                                                                                                                                                                                   0.0s
 => CACHED [2/3] WORKDIR /var/task                                                                                                                                                                                                  0.0s
 => CACHED [3/3] COPY ./ ./                                                                                                                                                                                                         0.0s
 => exporting to image                                                                                                                                                                                                              0.0s
 => => exporting layers                                                                                                                                                                                                             0.0s
 => => writing image sha256:e26aedd62dcec294fd7314bed8482bb9073623c1c2ff2a005eb6ac1bb6740d36                                                                                                                                        0.0s
 => => naming to docker.io/library/aws-lambda-local:testing                                                                                                                                                                         0.0s
docker: Error response from daemon: Conflict. The container name "/remainingtimedefault" is already in use by container "0a1ac6973c0b3af7da123e9b4b83c15a94030dd64b51a224ad38a416bd082443". You have to remove (or rename) that container to be able to reuse that name.
See 'docker run --help'.
.docker: Error response from daemon: Conflict. The container name "/remainingtimeten" is already in use by container "e33c68dd37c9e05bf78dc5a9ac4c717dd4dff152d8253c906c7ad39679796654". You have to remove (or rename) that container to be able to reuse that name.
See 'docker run --help'.
.docker: Error response from daemon: Conflict. The container name "/remainingtimethree" is already in use by container "635983bf9fe8d2eaef56cb90ed5a1716992fd883da68d62bacffc42e81f534ca". You have to remove (or rename) that container to be able to reuse that name.
See 'docker run --help'.
.6e0a5ba623f331fcd11891836d395c7a899813b45031e3079470c9d8b17236b4
.43040af058d655d5a5953e6c55dedf69f83b2f09ec7e45243757371c377a6779
.b640cd2ee547ab8745cefb0abaad543c58cfd7ee750220207234c08858f5b0ea
.docker: Error response from daemon: Conflict. The container name "/testing" is already in use by container "b640cd2ee547ab8745cefb0abaad543c58cfd7ee750220207234c08858f5b0ea". You have to remove (or rename) that container to be able to reuse that name.
See 'docker run --help'.
.fc8077999ffa686ccec9a5f940d1c0ccdb44bab8eb0f43da0db7d030fc3b976e
.docker: Error response from daemon: Conflict. The container name "/testing" is already in use by container "b640cd2ee547ab8745cefb0abaad543c58cfd7ee750220207234c08858f5b0ea". You have to remove (or rename) that container to be able to reuse that name.
See 'docker run --help'.
.envvarcheck
testing
timeout
exception
Error response from daemon: conflict: unable to remove repository reference "aws-lambda-local:testing" (must force) - container 635983bf9fe8 is using its referenced image e26aedd62dce
[+] Building 0.4s (8/8) FINISHED
 => [internal] load build definition from Dockerfile-python36                                                                                                                                                                       0.0s
 => => transferring dockerfile: 217B                                                                                                                                                                                                0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                   0.0s
 => => transferring context: 2B                                                                                                                                                                                                     0.0s
 => [internal] load metadata for public.ecr.aws/lambda/python:3.8                                                                                                                                                                   0.3s
 => [1/3] FROM public.ecr.aws/lambda/python:3.8@sha256:af98ec7625cc27f3efb189a753f57be48d4bedf4f558770157f3651e8bcb56a6                                                                                                             0.0s
 => [internal] load build context                                                                                                                                                                                                   0.0s
 => => transferring context: 107B                                                                                                                                                                                                   0.0s
 => CACHED [2/3] WORKDIR /var/task                                                                                                                                                                                                  0.0s
 => CACHED [3/3] COPY ./ ./                                                                                                                                                                                                         0.0s
 => exporting to image                                                                                                                                                                                                              0.0s
 => => exporting layers                                                                                                                                                                                                             0.0s
 => => writing image sha256:e26aedd62dcec294fd7314bed8482bb9073623c1c2ff2a005eb6ac1bb6740d36                                                                                                                                        0.0s
 => => naming to docker.io/library/aws-lambda-local:testing-py36                                                                                                                                                                    0.0s
45bb379b15c6edec3bfc81336e389c1e5224c5ee221219fcdd6706068c6168dc
.1bf6581903f9aa1a0852c73f5f6788fa5856de21034136fe701cc8ff7521ae3f
.testing
assert-overwritten
Untagged: aws-lambda-local:testing-py36

----------------------------------------------------------------------
Ran 11 tests in 35.430s

OK

The text was updated successfully, but these errors were encountered:

The integration tests were failing due to multiple issues. Multiple tests were using the same container name (`testing`), which meant some of the tests weren't running. Now each container has a unique name. The same port (9000) was being used for multiple tests. This wasn't showing as an issue as these tests also used the same container name. The ports are now unique per test, with them being number sequentially to avoid any conflicts. Only some of the running containers were terminated when the suite finished executing. This caused many tests to not run properly on subsequent invocation of the suite. Now all containers are terminated when the tests complete. The timeout tests weren't waiting for the container be ready. This was caused by the order of operations being wrong, with the sleep step occurring after the request to the endpoint, not before. This commit fixes aws#47.

valerena · 2021-10-04T14:13:46Z

Fixed by the linked PR. Let us know if you're still experiencing these issues.

skwashd mentioned this issue Sep 4, 2021

Fix integration tests #48

Closed

skwashd changed the title ~~Test suite is broken~~ Integration test suite is broken Sep 5, 2021

valerena linked a pull request Oct 4, 2021 that will close this issue

Support for arm64 architecture #52

Merged

valerena closed this as completed Oct 4, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Integration test suite is broken #47

Integration test suite is broken #47

skwashd commented Sep 4, 2021

valerena commented Oct 4, 2021

Integration test suite is broken #47

Integration test suite is broken #47

Comments

skwashd commented Sep 4, 2021

First Invocation

Second Invocation

valerena commented Oct 4, 2021