Elixir projects always recompile / Dialyzer cannot open files #5664

conradwt · 2017-01-17T05:03:33Z

TL;DR by @josevalim - This is an issue with zlib 1.2.9 and Erlang. To fix it, downgrade zlib or upgrade to latest Erlang possible.

Environment

macOS 10.12.2 (16C67)
Elixir 1.5.0-dev (b7bc5ae)
Erlang 19.1_0+hipe+ssl+wxwidgets (built and installed through MacPorts)
Zlib 1.2.11_0

Current behavior

I'm seeing the following test failure when performing the following command:

$ make clean test

  1) Kernel.DialyzerTest: failure on setup_all callback, test invalidated
     ** (throw) {:dialyzer_error, [67, 111, 117, 108, 100, 32, 110, 111, 116, 32, 103, 101, 116, 32, 97, 98, 115, 116, 114, 97, 99, 116, 32, 99, 111, 100, 101, 32, 102, 111, 114, 32, 102, 105, 108, 101, 58, 32, '/Users/conradwt/elixir.dir/projects/elixir/bin/../lib/elixir/ebin/Elixir.Kernel.beam', 32, 40, 112, 108, 101, 97, 115, 101, 32, 114, ...]}
     stacktrace:
       dialyzer_plt.erl:469: :dialyzer_plt.compute_md5_from_file/1
       dialyzer_plt.erl:458: :dialyzer_plt."-compute_md5_from_files/1-lc$^0/1-0-"/1
       dialyzer_plt.erl:458: :dialyzer_plt."-compute_md5_from_files/1-lc$^0/1-0-"/1
       dialyzer_plt.erl:458: :dialyzer_plt.compute_md5_from_files/1
       dialyzer_cl.erl:74: :dialyzer_cl.build_plt/1
       dialyzer.erl:167: :dialyzer.run/1
       test/elixir/kernel/dialyzer_test.exs:27: Kernel.DialyzerTest.__ex_unit_setup_all_0/1
       test/elixir/kernel/dialyzer_test.exs:3: Kernel.DialyzerTest.__ex_unit__/2

Finished in 11.8 seconds (7.6s on load, 4.2s on tests)
3453 tests, 1 failure, 7 invalid

Expected behavior

I would expect zero test failures.

The text was updated successfully, but these errors were encountered:

josevalim · 2017-01-17T14:39:40Z

I cannot reproduce this failure. Have you attempted another Erlang version to see if this is something specific to dialyzer in this Erlang version? I have used 18 and 19.0 and 19.2.

conradwt · 2017-01-17T16:05:25Z

@josevalim Is Dialyzer being included within the latest builds of Elixir? BTW, I just started seeing this issue with Erlang 19.1. I'll try testing against Erlang 19.2.

josevalim · 2017-01-17T17:18:38Z

@josevalim Is Dialyzer being included within the latest builds of Elixir?

What do you mean? Dialyzer ships with Erlang and we don't really do anything special, we just use it slightly in our test suite to check some code won't emit dialyzer warnings.

fishcakez · 2017-01-17T17:26:10Z

I have improved the error when dialyzer fails in 4cfe280. Unfortunately we don't know why dialyzer failed to fetch the abstract code here (thats the error).

conradwt · 2017-01-17T19:12:26Z

@josevalim I didn't know if Dialyzer was being enabled by default within Erlang or Elixir. It could very well be an issue with Macports build of Erlang. Now, I'll try building from source and report back here.

fishcakez · 2017-01-17T19:52:29Z

@conradwt dialyzer is included with Erlang (given the stacktrace it's there). However we can't debug why it doesn't work without more information. Perhaps Elixir isn't build with debug info? Do we have a way to do that?

conradwt · 2017-01-17T20:00:31Z

@fishcakez I'm just doing the following to build Elixir:

$ git clone https://github.com/elixir-lang/elixir.git
$ cd elixir
$ make clean test

fishcakez · 2017-01-17T20:42:17Z

@conradwt I am unsure what to suggest.

We could see what happens if you do:

$ bin/iex
iex 1> :beam_lib.chunks(:code.which(Kernel),[:abstract_code])

conradwt · 2017-01-17T21:02:17Z

@fishcakez Here's the output:

iex(1)> :beam_lib.chunks(:code.which(Kernel),[:abstract_code])
{:error, :beam_lib,
 {:invalid_chunk,
  '/Users/conradwt/elixir.dir/projects/elixir/bin/../lib/elixir/ebin/Elixir.Kernel.beam',
  'Abst'}}

fishcakez · 2017-01-17T21:28:55Z

@conradwt ah, not sure whats going on here. Lets see [:compile_info] instead of [:abstract_code].

conradwt · 2017-01-18T00:43:27Z

@fishcakez Here's the output using [:compile_info]:

iex(1)> :beam_lib.chunks(:code.which(Kernel),[:compile_info]) 
{:ok,
 {Kernel,
  [compile_info: [options: [:debug_info], version: '7.0.2',
    source: '/Users/conradwt/elixir.dir/projects/elixir/lib/elixir/lib/kernel.ex']]}}
iex(2)>

fishcakez · 2017-01-18T13:01:58Z

Interesting, Kernel was compiled on ERTS-7.0.2 which is OTP-18.0.2. I wonder if Elixir been compiled on 18.0.2 and the abstract code is not valid on OTP-19.1 (the version listed at the top). Could you try again with Elixir compiled and run on the same major version of OTP?

fishcakez · 2017-01-18T13:25:56Z

I could not reproduce this when I built on OTP-18.0.3 and ran tests with OTP-19.1. I did not have OTP-18.0.2 to hand unfortunately.

josevalim · 2017-01-20T23:49:54Z

@conradwt can you please try to run make clean test and let us know if it works? It seems the Kernel module compiled in your Elixir checkout was compiled quite some time ago OR it was compiled recently but with a different Erlang version. Please make sure you can compile from scratch with the same Erlang version.

fishcakez · 2017-01-21T20:46:28Z

I wonder if its http://erlang.org/pipermail/erlang-bugs/2017-January/005206.html?

conradwt · 2017-01-21T21:22:12Z

@fishcakez This appears to be the same issue. Furthermore, OpenSSL depends on Zlib and Erlang has a dependency on OpenSSL.

conradwt · 2017-01-22T01:03:13Z

@josevalim It appears that there maybe an issue with the way the Erlang 19.1 port has been built because I was able to build Erlang 19.2 from source using kerl and Elixir from source with zero test failures.

josevalim · 2017-01-22T11:16:23Z

Interesting. So it does seems to be a bug elsewhere. How was your earlier 19.1 built?

manukall · 2017-01-25T09:22:55Z

i'm running into the same error. also on arch linux. i've tried with erlang 19.0 and erlang 19.2, both built with asdf.

it's not really a problem for me, but if i can help, please let me know.

manukall · 2017-01-25T09:51:21Z

turns out it actually is a problem, and it's also pretty clear what is causing it:

as of http://erlang.org/pipermail/erlang-questions/2017-January/091452.html, binary_to_term is having problems with zlib >= v1.2.9. this also affects mix compile, as the manifest can not be parsed. that was actually the reason why i built elixir in the first place: the whole project get's recompiled with every request in phoenix.

i'm downgrading zlib for now, but i suspect this will be a problem when distributions with slower release cycles upgrade their zlib.

josevalim · 2017-01-25T11:08:17Z

@manukall excellent catch. We will keep our eyes open as this will certainly cause further issues down the road.

conradwt · 2017-01-27T00:01:27Z

@josevalim My earlier build of Erlang 19.1 was built through MacPorts.

fishcakez · 2017-01-27T12:52:51Z

@conradwt the elixir team doesn't maintain elixir packages, so you would need to contact whoever maintains the package for the relevant package manager to get it fixed.

dvracar · 2017-01-27T13:28:57Z

I can confirm this on my archlinux installation too. Downgraded to zlib 1.2.8 and recompile problem disappeared...

Eiji7 · 2017-01-31T11:41:57Z

@conradwt, @josevalim I confirmed this when launched my script to auto clone and recompile 3 Elixir versions after my distro update. I want to create new issue, but found it, so I copied my post. Here you can find more detailed info about it:

Environment

Elixir & Erlang versions (elixir --version):

Erlang/OTP 19 [erts-8.1] [source] [64-bit] [smp:4:4] [async-threads:10]

I tried to execute make clean test on: tags/v1.4.1, v1.4 and master.

Operating system:
Funtoo Linux (Gentoo based) Intel64-nehalem funtoo-stable standard from stage 1
Note: I installed erlang package from official distribution repository and I don't have installed Elixir (I just compile from git).

[ebuild   R    ] dev-lang/erlang-19.1::gentoo  USE="smp ssl tk wxwidgets -compat-ethread -dirty-schedulers -doc -emacs -hipe -java -kpoll (-libressl) -odbc -sctp (-systemd)" 0 KiB
[ebuild   R    ] sys-libs/zlib-1.2.11::gentoo  USE="minizip -static-libs" ABI_X86="32 (64) (-x32)" 0 KiB
[ebuild  N     ] dev-lang/elixir-1.3.0-r1::gentoo  1704 KiB

You can find a output of emerge --info here.

Current behavior

On: elixir (exunit) make job I got 1 test failure:

tags/v1.4.1 - the complete build log is located at pastebin.

  1) Kernel.DialyzerTest: failure on setup_all callback, test invalidated
     ** (throw) {:dialyzer_error, [67, 111, 117, 108, 100, 32, 110, 111, 116, 32, 103, 101, 116, 32, 97, 98, 115, 116, 114, 97, 99, 116, 32, 99, 111, 100, 101, 32, 102, 111, 114, 32, 102, 105, 108, 101, 58, 32, '/home/eiji/elixir-releases/tags/v1.4.1/bin/../lib/elixir/ebin/Elixir.Kernel.beam', 32, 40, 112, 108, 101, 97, 115, 101, 32, 114, ...]}
     stacktrace:
       dialyzer_plt.erl:469: :dialyzer_plt.compute_md5_from_file/1
       dialyzer_plt.erl:458: :dialyzer_plt."-compute_md5_from_files/1-lc$^0/1-0-"/1
       dialyzer_plt.erl:458: :dialyzer_plt."-compute_md5_from_files/1-lc$^0/1-0-"/1
       dialyzer_plt.erl:458: :dialyzer_plt.compute_md5_from_files/1
       dialyzer_cl.erl:74: :dialyzer_cl.build_plt/1
       dialyzer.erl:167: :dialyzer.run/1
       test/elixir/kernel/dialyzer_test.exs:27: Kernel.DialyzerTest.__ex_unit_setup_all_0/1
       test/elixir/kernel/dialyzer_test.exs:3: Kernel.DialyzerTest.__ex_unit__/2

v1.4- the complete build log is located at pastebin.

  1) Kernel.DialyzerTest: failure on setup_all callback, test invalidated
     ** (throw) {:dialyzer_error, [67, 111, 117, 108, 100, 32, 110, 111, 116, 32, 103, 101, 116, 32, 97, 98, 115, 116, 114, 97, 99, 116, 32, 99, 111, 100, 101, 32, 102, 111, 114, 32, 102, 105, 108, 101, 58, 32, '/home/eiji/elixir-releases/branches/v1.4/bin/../lib/elixir/ebin/Elixir.Kernel.beam', 32, 40, 112, 108, 101, 97, 115, 101, 32, 114, ...]}
     stacktrace:
       dialyzer_plt.erl:469: :dialyzer_plt.compute_md5_from_file/1
       dialyzer_plt.erl:458: :dialyzer_plt."-compute_md5_from_files/1-lc$^0/1-0-"/1
       dialyzer_plt.erl:458: :dialyzer_plt."-compute_md5_from_files/1-lc$^0/1-0-"/1
       dialyzer_plt.erl:458: :dialyzer_plt.compute_md5_from_files/1
       dialyzer_cl.erl:74: :dialyzer_cl.build_plt/1
       dialyzer.erl:167: :dialyzer.run/1
       test/elixir/kernel/dialyzer_test.exs:27: Kernel.DialyzerTest.__ex_unit_setup_all_0/1
       test/elixir/kernel/dialyzer_test.exs:3: Kernel.DialyzerTest.__ex_unit__/2

master - the complete build log is located at pastebin.

  1) Kernel.DialyzerTest: failure on setup_all callback, test invalidated
     ** (RuntimeError) dialyzer error: Could not get abstract code for file: /home/eiji/elixir-releases/branches/master/bin/../lib/elixir/ebin/Elixir.Kernel.beam (please recompile it with +debug_info)
     
     stacktrace:
       test/elixir/kernel/dialyzer_test.exs:115: Kernel.DialyzerTest.dialyzer_run/1
       test/elixir/kernel/dialyzer_test.exs:27: Kernel.DialyzerTest.__ex_unit_setup_all_0/1
       test/elixir/kernel/dialyzer_test.exs:3: Kernel.DialyzerTest.__ex_unit__/2

Expected behavior

All tests should pass.

btw. There are lots of deprecation warnings especially at master.

I don't have other zlib package versions in my repo :/

josevalim · 2017-01-31T12:00:19Z

Thanks. Hopefully a new Erlang version will zlib fixes will be out soon.

fishcakez · 2017-02-02T13:13:49Z

A fix is at erlang/otp@e271199. This has been included in OTP-18.3.4.5.

Eiji7 · 2017-02-05T16:10:01Z

@fishcakez: Is there any plans for other Erlang releases (especially >19)? I compiled Erlang from evm (version OTP_19.2) and still have same problem :/

fishcakez · 2017-02-05T16:45:33Z

@Eiji7 the commit I linked is in the maint branch so will appear in the next OTP-19 release. I will reply when that happens.

Eiji7 · 2017-02-05T16:46:53Z

@fishcakez: ok, thank you!

Eiji7 · 2017-02-08T11:03:41Z

Workarounds for Gentoo/Funtoo users:

You could download older .ebuild zlib package and create your local repository (it's already well documented) + mask newer versions of zlib

Execute following commands:

# compile openssl without zlib support
USE="-zlib" emerge openssl
# temporary unmerge zlib package
emerge --unmerge zlib
evm uninstall 19.2 # if you have already compiled
evm install 19.2 # Erlang Version Manager
evm default 19.2 # select default Erlang version
cd directory/path/for/elixir/source
make test clean
emerge -av zlib

fishcakez · 2017-02-08T11:11:40Z

OTP-19.2.3 was released which includes the fix erlang/otp@e271199

Eiji7 · 2017-02-08T17:59:46Z

@conradwt, @fishcakez and @josevalim:
Erlang 19.2.3 and Elixir 1.4.1 compiled from source without errors.
CONFIRMED
😄

josevalim · 2017-03-27T11:05:11Z

Erlang 19.3 is also out with a fix for this. We can finally close this. :)

josevalim added App:Elixir (stdlib) Kind:Bug Note:Needs more info labels Jan 18, 2017

josevalim changed the title ~~Kernel.DialyzerTest: failure on setup_all callback, test invalidated~~ Dialyzer cannot open files / Elixir projects always recompile Jan 25, 2017

josevalim changed the title ~~Dialyzer cannot open files / Elixir projects always recompile~~ Elixir projects always recompile / Dialyzer cannot open files Jan 25, 2017

josevalim added Note:Erlang/OTP Note:Open for guidance and removed App:Elixir (stdlib) Kind:Bug labels Jan 25, 2017

josevalim removed the Note:Needs more info label Jan 25, 2017

Eiji7 mentioned this issue Feb 1, 2017

Send test fails to stderr instead of stdout #5726

Closed

josevalim mentioned this issue Feb 2, 2017

Kernel.DialyzerTest: failure on setup_all callback, tests invalidated #5163

Closed

fishcakez mentioned this issue Feb 12, 2017

mix dialyze fails with an error about erlang.beam fishcakez/dialyze#17

Closed

josevalim closed this as completed Mar 27, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Elixir projects always recompile / Dialyzer cannot open files #5664

Elixir projects always recompile / Dialyzer cannot open files #5664

conradwt commented Jan 17, 2017 •

edited

Loading

josevalim commented Jan 17, 2017

conradwt commented Jan 17, 2017

josevalim commented Jan 17, 2017

fishcakez commented Jan 17, 2017

conradwt commented Jan 17, 2017

fishcakez commented Jan 17, 2017

conradwt commented Jan 17, 2017 •

edited

Loading

fishcakez commented Jan 17, 2017

conradwt commented Jan 17, 2017

fishcakez commented Jan 17, 2017

conradwt commented Jan 18, 2017

fishcakez commented Jan 18, 2017

fishcakez commented Jan 18, 2017

josevalim commented Jan 20, 2017

fishcakez commented Jan 21, 2017

conradwt commented Jan 21, 2017

conradwt commented Jan 22, 2017 •

edited

Loading

josevalim commented Jan 22, 2017

manukall commented Jan 25, 2017

manukall commented Jan 25, 2017

josevalim commented Jan 25, 2017

conradwt commented Jan 27, 2017

fishcakez commented Jan 27, 2017

dvracar commented Jan 27, 2017

Eiji7 commented Jan 31, 2017 •

edited

Loading

josevalim commented Jan 31, 2017

fishcakez commented Feb 2, 2017

Eiji7 commented Feb 5, 2017 •

edited

Loading

fishcakez commented Feb 5, 2017

Eiji7 commented Feb 5, 2017

Eiji7 commented Feb 8, 2017

fishcakez commented Feb 8, 2017

Eiji7 commented Feb 8, 2017 •

edited

Loading

josevalim commented Mar 27, 2017

Elixir projects always recompile / Dialyzer cannot open files #5664

Elixir projects always recompile / Dialyzer cannot open files #5664

Comments

conradwt commented Jan 17, 2017 • edited Loading

Environment

Current behavior

Expected behavior

josevalim commented Jan 17, 2017

conradwt commented Jan 17, 2017

josevalim commented Jan 17, 2017

fishcakez commented Jan 17, 2017

conradwt commented Jan 17, 2017

fishcakez commented Jan 17, 2017

conradwt commented Jan 17, 2017 • edited Loading

fishcakez commented Jan 17, 2017

conradwt commented Jan 17, 2017

fishcakez commented Jan 17, 2017

conradwt commented Jan 18, 2017

fishcakez commented Jan 18, 2017

fishcakez commented Jan 18, 2017

josevalim commented Jan 20, 2017

fishcakez commented Jan 21, 2017

conradwt commented Jan 21, 2017

conradwt commented Jan 22, 2017 • edited Loading

josevalim commented Jan 22, 2017

manukall commented Jan 25, 2017

manukall commented Jan 25, 2017

josevalim commented Jan 25, 2017

conradwt commented Jan 27, 2017

fishcakez commented Jan 27, 2017

dvracar commented Jan 27, 2017

Eiji7 commented Jan 31, 2017 • edited Loading

Environment

Current behavior

Expected behavior

josevalim commented Jan 31, 2017

fishcakez commented Feb 2, 2017

Eiji7 commented Feb 5, 2017 • edited Loading

fishcakez commented Feb 5, 2017

Eiji7 commented Feb 5, 2017

Eiji7 commented Feb 8, 2017

fishcakez commented Feb 8, 2017

Eiji7 commented Feb 8, 2017 • edited Loading

josevalim commented Mar 27, 2017

conradwt commented Jan 17, 2017 •

edited

Loading

conradwt commented Jan 17, 2017 •

edited

Loading

conradwt commented Jan 22, 2017 •

edited

Loading

Eiji7 commented Jan 31, 2017 •

edited

Loading

Eiji7 commented Feb 5, 2017 •

edited

Loading

Eiji7 commented Feb 8, 2017 •

edited

Loading