terminal_output

[djarecka]

Summary

A question on neurostar motivated me to check terminal_output and I'm not sure if I understand how this should work. For example, I set terminal_output="file" for node with BET and I can see that the file output.nipype is created, but I it's removed before the node ends. In addition, I think the file is empty before removing anyway and the output is sent to my terminal.

Can someone please explain me what is the expected behavior.

[satra]

the file is removed because it's not needed downstream from a node. if you run it as an interface the file will stay. BET i don't think produces any stdout

the only files kept by a node are listed here:
https://github.com/nipy/nipype/blob/master/nipype/pipeline/engine/utils.py#L1411

any stdout/error is available via the runtime field of the result from an execution of an interface/node

[djarecka]

So this is not a bug and terminal_output is not meant to save the output/error for later references?

So this will not help with this qst?

[satra]

@djarecka - that is currently true. however there are some interfaces like fsl.ImageStats, where we interpret the terminal output into an output key.

we could consider whether runtime information more generally can be used as an input to something else, but that is something we currently don't support.

however i think the information from a node can be suppressed (which i think is the question) in different ways:

1. changing logging levels (this will suppress everything)
2. for spm, unfortunately this comes from the matlab interface: https://github.com/nipy/nipype/blob/master/nipype/interfaces/matlab.py#L161

which defaults to allatonce. this would need to be configurable to use file instead.

[spanta28]

Hi Satra, thanks for the reply. So I am using spm12 standalone and matlab mcr.
I changed my /opt/miniconda/envs/default/lib/python3.5/site-packages/nipype/interfaces/matlab.py
def _run_interface(self, runtime):
self.terminal_output = 'file'

Then it gives me the foll. error:
Python 3.5.1 | packaged by conda-forge | (default, Jun 29 2016, 12:18:14)
[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)] on linux
Type "help", "copyright", "credits" or "license" for more information.

from nipype.interfaces import spm
spm.SPMCommand.set_mlab_paths(matlab_cmd='/opt/spm12/run_spm12.sh /opt/mcr/v92 script', use_mcr=True)
Traceback (most recent call last):
File "", line 1, in
File "/opt/miniconda/envs/default/lib/python3.5/site-packages/nipype/interfaces/spm/base.py", line 316, in set_mlab_paths
use_mcr=use_mcr)
File "/opt/miniconda/envs/default/lib/python3.5/site-packages/nipype/interfaces/spm/base.py", line 231, in getinfo
out = sd._strip_header(out.runtime.stdout)
File "/opt/miniconda/envs/default/lib/python3.5/site-packages/nipype/utils/spm_docs.py", line 53, in _strip_header
index = doc.index(hdr)
ValueError: substring not found

My spm12 standalone and mcr is running fine with the 'allatonce' option, but prints things to stdout.
Let me know what I can do here. Also for 1) let me know how I can change the logging levels, that would be helpful.

Thanks

[satra]

@spanta28 - that may be a bug - @djarecka - could you take a look at this within the nipype tutorial container?

regarding logging: http://nipype.readthedocs.io/en/latest/users/config_file.html#enabling-logging-to-file

also look at the debug section. if you set logging to CRITICAL only errors will be posted.

[djarecka]

@satra - just confirming that I can see a similar error after changing self.terminal_output.

[spanta28]

@satra I tried the logging acc. to
http://nipype.readthedocs.io/en/latest/users/config_file.html#enabling-logging-to-file
Still its printing out nipype outputs as before to stdout

[spanta28]

I will try the debug CRITICAL thing now

[spanta28]

This worked for me in suppressing nipype.workflow to stdout
from nipype import logging
logging.getLogger('nipype.workflow').setLevel('CRITICAL')

Thanks @satra

[djarecka]

I've been debugging the ValueError, it looks like that results["stdout"] reads stream only for allatonce, otherwise it's empty.
At the end strip_header is called with an empty string and this gives the error.

@satra - any suggestions what should be changed? Is it ok that results["stdout"] is empty and I only should change strip_header, so it works for an empty string?

[satra]

@djarecka - perhaps write a few tests with a shell script that write out to both stdout and stderr and see that all the terminal_output options return the appropriate values.

[KamalakerDadi]

Hi Nipype team,

Just a follow up on this issue

======================================================================
ERROR: prepare dir containing fake Matlab and SPM installs and launch tests.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/circleci/.local/lib/python3.5/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/circleci/project/pypreprocess/tests/test_configure_spm.py", line 327, in test_spm_config
    _execute_spm_config_test(defaults, explicitly_set, spm_root)
  File "/home/circleci/project/pypreprocess/tests/test_configure_spm.py", line 260, in _execute_spm_config_test
    spm_dir = _configure_spm(defaults=defaults, **explicitly_set)
  File "/home/circleci/project/pypreprocess/configure_spm.py", line 659, in _configure_spm
    choice['configure'](*found)
  File "/home/circleci/project/pypreprocess/configure_spm.py", line 499, in _configure_spm_using_mcr
    matlab_cmd='{} batch'.format(spm_mcr), use_mcr=True)
  File "/home/circleci/.local/lib/python3.5/site-packages/nipype/interfaces/spm/base.py", line 316, in set_mlab_paths
    use_mcr=use_mcr)
  File "/home/circleci/.local/lib/python3.5/site-packages/nipype/interfaces/spm/base.py", line 231, in getinfo
    out = sd._strip_header(out.runtime.stdout)
  File "/home/circleci/.local/lib/python3.5/site-packages/nipype/utils/spm_docs.py", line 53, in _strip_header
    index = doc.index(hdr)
ValueError: substring not found

This seems to appear with recent Nipype version. Any ideas why this happens and temporary fix for this ?

is this something changed we are missing in using between 'batch' and 'run script' ?

Thanks a lot for your time on this.

[effigies]

What code produces this issue? It looks like you're running nose, which is not supported. Tests should be run via pytest.

[KamalakerDadi]

Thanks @effigies we missed this details. Yes we are running nosetests in pypreprocess.

[KamalakerDadi]

from which version is it stopped supporting nose ?

I am trying to look at the changes but could not find it.
https://nipype.readthedocs.io/en/latest/changes.html

[effigies]

0.13.0-rc1

nipype/doc/changelog/0.X.X-changelog

Line 145 in 37f3781

* TST: Replace nose and unittest with pytest (https://github.com/nipy/nipype/pull/1722, https://github.com/nipy/nipype/pull/1751)

Looks like the changelog on the site didn't get updated when we split 0.x changes from 1.x (the file was getting very large).

[kchawla-pi]

Issue #2690 and PR #2691 deals with the error.
It shows up when I use pytest as well.

[djarecka]

@kchawla-pi - I'm not sure if the original issue is the same as #2690.

But could you please be more specific about error from pytest. Is it with one of a nipype tests?

[kchawla-pi]

Apologies, I was referring to the following comments, when I raised issue #2690
#2640 (comment)
#2640 (comment)

[djarecka]

no reason to apologize, thank you for reporting! I guess there are a few problems described in this issue. Should finish it!

But just coming back to my question - did you have the error when running one of our test?

[kchawla-pi]

Nope, when I was running one of ours.
from https://github.com/neurospin/pypreprocess:
pypreprocess/examples/easy_start/nipype_preproc_spm_auditory.py

[effigies]

It's unclear to me whether this issue has been fully solved. I'm going to close it. If there is still some remaining issue to address, let's open a new thread, so it's easier to read.

[spanta28]

Its resolved. Sorry for late reply.