Flakey tests in Gentoo Linux

[hololeap]

I am getting very mixed results from the tests for hls-rename-plugin-1.0.1.0. In three different runs, I got three different results.

Run 1

1. FAIL (6.61s)
       WorkspaceEdit contains neither documentChanges nor changes!
       CallStack (from HasCallStack):
         error, called at src/Language/LSP/Test/Session.hs:368:9 in lsp-test-0.14.1.0-KE6O6LMWm63H2RWAlrww2P:Language.LSP.Test.Session
       Use -p '/Import hiding/' to rerun this test only.
   

2. FAIL (6.51s)
       Received an expected error in a response for id SomeLspId (IdInt 2):
       ResponseError {_code = InternalError, _message = "No AST for file: NormalizedFilePath \"/var/tmp/portage/dev-haskell/hls-rename-plugin-1.0.1.0/work/hls-rename-plugin-1.0.1.0/test/testdata/LetExpression.hs\"", _xdata = Nothing}
       Use -p '/Let expression/' to rerun this test only.
   

3. FAIL (6.35s)
       Received an expected error in a response for id SomeLspId (IdInt 2):
       ResponseError {_code = InternalError, _message = "No AST for file: NormalizedFilePath \"/var/tmp/portage/dev-haskell/hls-rename-plugin-1.0.1.0/work/hls-rename-plugin-1.0.1.0/test/testdata/QualifiedAs.hs\"", _xdata = Nothing}
       Use -p '/Qualified as/' to rerun this test only.
   

4. FAIL (6.97s)
   2022-09-24T18:20:48.758605Z | Info | Logging heap statistics every 60.00s
       Received an expected error in a response for id SomeLspId (IdInt 2):
       ResponseError {_code = InternalError, _message = "No AST for file: NormalizedFilePath \"/var/tmp/portage/dev-haskell/hls-rename-plugin-1.0.1.0/work/hls-rename-plugin-1.0.1.0/test/testdata/QualifiedShadowing.hs\"", _xdata = Nothing}
       Use -p '/Qualified shadowing/' to rerun this test only.
   

5. FAIL (7.16s)
       Received an expected error in a response for id SomeLspId (IdInt 2):
       ResponseError {_code = InternalError, _message = "No AST for file: NormalizedFilePath \"/var/tmp/portage/dev-haskell/hls-rename-plugin-1.0.1.0/work/hls-rename-plugin-1.0.1.0/test/testdata/QualifiedFunction.hs\"", _xdata = Nothing}
       Use -p '/Qualified function/' to rerun this test only.
   

5 out of 20 tests failed (27.80s)

Run 2

All 20 tests passed (26.18s)

Run 3

1. FAIL (6.22s)
       Test output was different from 'test/testdata/QualifiedFunction.expected.hs'. Output of ["git","-c","core.fileMode=false","diff","--no-index","--text","--exit-code","test/testdata/QualifiedFunction.expected.hs","/var/tmp/portage/dev-haskell/hls-rename-plugin-1.0.1.0/temp/QualifiedFunction.expected65-81.actual"]:
       
       Use -p '/Qualified function/' to rerun this test only.
   

1 out of 20 tests failed (26.31s)

---

Full build logs: #1 #2 #3

---

Your environment

GHC 9.0.2 on Gentoo Linux. All packages are installed through the system package manager.

---

Other gentoo-haskell-related test issues:

• Multiple test failures in Gentoo's network sandbox #3126
• hls-hlint-plugin test failure in Gentoo Linux #3217
• hls-gadt-plugin test failures in Gentoo Linux #3218
• ghcide test failures in Gentoo Linux #3221

[hololeap]

Something similar with hls-splice-plugin-1.0.2.0:

I was only able to reproduce this twice in ten runs, but it seems like it is probably related.

---

Run 1

1. FAIL (6.46s)
         Test output was different from 'test/testdata/TQQPat.expected.hs'. Output of ["git","-c","core.fileMode=false","diff","--no-index","--text","--exit-code","test/testdata/TQQPat.expected.hs","/var/tmp/portage/dev-haskell/hls-splice-plugin-1.0.2.0/temp/TQQPat.expected65-121.actual"]:
         
         Use -p '$0=="splice.Pattern Splices.TQQPat (golden)"' to rerun this test only.
   

1 out of 31 tests failed (51.22s)

Run 10

1. FAIL (7.49s)
         Test output was different from 'test/testdata/TQQPatError.expected.hs'. Output of ["git","-c","core.fileMode=false","diff","--no-index","--text","--exit-code","test/testdata/TQQPatError.expected.hs","/var/tmp/portage/dev-haskell/hls-splice-plugin-1.0.2.0/temp/TQQPatError.expected65-127.actual"]:
         
         Use -p '$0=="splice.Pattern Splices.TQQPatError (golden)"' to rerun this test only.
   

1 out of 31 tests failed (54.41s)

---

Full build logs:

• Failing #1
• Failing #10
• Passing #2

[pepeiborra]

The problem is that Tasty runs test in parallel by default, and most HLS test suites do not handle this well. Set TASTY_NUM_THREADS to 1

[hololeap]

@pepeiborra

Thanks, changing the number of tasty threads to 1 seems to have fixed this batch.