Skip to content

Add minio container to mock s3 locally #597

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Feb 9, 2020
Merged

Add minio container to mock s3 locally #597

merged 3 commits into from
Feb 9, 2020

Conversation

yvrhdn
Copy link

@yvrhdn yvrhdn commented Feb 7, 2020

Extends docker-compose.yml to also launch an container with minio, backed by a persistent volume.
Closes #463

It seems to work, but I can't test it reliably due this other issue I'm having right now: #594 (comment)

@yvrhdn
Copy link
Author

yvrhdn commented Feb 8, 2020

TODO:

  • automatically create the bucket rust-docs-rs

Open questions:

  • Should we expose port 9000? This serves a webpage on which you can browse through the buckets/files.
    We don't expose the port of the database, so this would not be consistent.

  • Should we remove the code that can fallback to the database if S3 is not available?

Koenraad Verheyden added 2 commits February 8, 2020 01:01
docs.rs expects there to be a bucket 'rust-docs-rs', the minio Docker
image does not provide a nice way to launch with a default bucket.

As a workaround, we have to override the entrypoint to do some work
before starting the minio server. When the minio server is launched
with a directory that is not empty, it will interpret the directories
present as buckets it needs to create. So by adding an empty directory
'rust-docs-rs', we can trick minio in creating a similarly named
bucket.

This solution is a mixture of the following comments:
 - minio/minio#4769 (comment)
 - minio/minio#4769 (comment)
@yvrhdn yvrhdn marked this pull request as ready for review February 8, 2020 00:44
@jyn514
Copy link
Member

jyn514 commented Feb 8, 2020

Should we expose port 9000? This serves a webpage on which you can browse through the buckets/files.

You currently have it exposed. I kind of like this, I would find it very helpful for debugging.

Should we remove the code that can fallback to the database if S3 is not available?

That would make it harder to work on docs.rs without using docker ... Do you know if minio is easy to get set up on Windows?

@yvrhdn
Copy link
Author

yvrhdn commented Feb 8, 2020

That would make it harder to work on docs.rs without using docker ... Do you know if minio is easy to get set up on Windows?

I would imagine you could also use Docker on Windows?
There are instructions to install minio on your local machine, but I haven't tried this myself: https://docs.min.io/docs/minio-quickstart-guide.html

@jyn514
Copy link
Member

jyn514 commented Feb 8, 2020

Yes, but you can't run cargo test with docker.

@yvrhdn
Copy link
Author

yvrhdn commented Feb 8, 2020

Since port 9000 is open, you could run cargo test and let the tests connect to the minio instance from the docker-compose.
I have done something similar for tests that requires the database (you have to expose the port and set the correct environment variables).

But then we would also have to look at cleaning up the bucket after the tests have run...
--> seems like a good next PR: integrate S3/minio in test::TestEnvironment so we can test with S3Client as well :)

I think for now it is best to keep this code. If we find out no-one uses this fallback mechanism anymore we can clean it up.

@jyn514
Copy link
Member

jyn514 commented Feb 8, 2020

Wow, browsing s3 locally is super helpful. Thanks for working on this!

@jyn514
Copy link
Member

jyn514 commented Feb 8, 2020

Can you add instructions to the README for how to access this locally? Just 'navigate to localhost:9000` in your browser' is fine.

@yvrhdn
Copy link
Author

yvrhdn commented Feb 9, 2020

@jyn514 done! Rendered: https://github.com/koenaad/docs.rs/blob/mock_s3/README.md

@jyn514 jyn514 merged commit e131f00 into rust-lang:master Feb 9, 2020
@yvrhdn yvrhdn deleted the mock_s3 branch February 9, 2020 23:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Use minio instead of postgres locally
2 participants