Skip to content

Copy stream hangs if using gzip stream compression #499

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
uasan opened this issue Oct 7, 2022 · 4 comments
Open

Copy stream hangs if using gzip stream compression #499

uasan opened this issue Oct 7, 2022 · 4 comments
Labels
bug Something isn't working

Comments

@uasan
Copy link

uasan commented Oct 7, 2022

Hello, we caught a cool bug )

Streams work fine when copying, but if the stream is sent to the compression pipe, the entire pool of connections freezes.

Example, playback:

import postgres from 'postgres';
import { createGzip } from 'zlib';

const db = postgres({
  max: 2,
});

for (let i = 0; i < 10; i++) {
  let stream = await db
    .unsafe(`COPY pg_catalog.pg_attribute TO STDOUT`)
    .readable();

  stream.pipe(createGzip()).on('data', console.log);

  await db.unsafe(`SELECT $1 AS "CONTINUE"`, [i]);
}

console.log(await db.unsafe(`SELECT 'DONE' AS "END"`));

I play with pipe, but in any case, something is wrong with the connection pool after sending the stream to the pipe.

@uasan uasan changed the title Copy stream often hangs if using gzip stream compression Copy stream hangs if using gzip stream compression Oct 7, 2022
@uasan
Copy link
Author

uasan commented Oct 13, 2022

It seems that this problem was the cause of the connection errors that I described here #457. Due to the pipe, connections in the pool become busy in read mode, although postgres has returned the response and there is nothing to read

@porsager
Copy link
Owner

Sorry I haven't been on this @uasan .. I can repro here, so I hope I'll find some time to look into it soon!

@porsager porsager added the bug Something isn't working label Jun 26, 2023
@uasan
Copy link
Author

uasan commented Jul 4, 2023

I think this is due to the fact that the archiving stream sometimes pauses the stream

@Ivor808
Copy link

Ivor808 commented Jul 23, 2024

I wonder if I am hitting a similar issue. Certain copy commands are read to stdout into a readable stream. Those readable streams sometimes just hang and it seems as if the "end" event is never transmitted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants