Is `tox` test suite broken on travis.ci?

[paulperegud]

• py-solc Version: 3.2.0
• solc Version: 0.4.25
• Python Version: 3.6.6
• OS: linux

What was wrong?

Test suite seems to be broken. A lot of tests are in red. For example, test contracts defined in conftest.py produce warnings on newer versions of solc and that warning does not pass is_benign check.

All while tox passes tests on travis.ci somehow.

Example of failed pytest run. Command:
python -m pytest -k test_providing_stdin

It's output:

================================================================================= test session starts ==================================================================================
platform linux -- Python 3.6.6, pytest-3.8.2, py-1.7.0, pluggy-0.7.1 -- /home/pepesza/code/py-solc/venv/bin/python
cachedir: .pytest_cache
rootdir: /home/pepesza/code/py-solc, inifile: pytest.ini
collected 58 items / 57 deselected                                                                                                                                                     

tests/core/wrapper/test_solc_wrapper.py::test_providing_stdin FAILED                                                                                                             [100%]

======================================================================================= FAILURES =======================================================================================
_________________________________________________________________________________ test_providing_stdin _________________________________________________________________________________

FOO_SOURCE = 'pragma solidity ^0.4.17;\n\ncontract Foo {\n    function Foo() public {}\n\n    function return13() public pure returns (uint) {\n        return 13;\n    }\n}\n'

    def test_providing_stdin(FOO_SOURCE):
        output, err, _, _ = solc_wrapper(stdin=FOO_SOURCE, bin=True)
        assert output
        assert 'Foo' in output
>       assert is_benign(err)
E       assert False
E        +  where False = is_benign('<stdin>:4:5: Warning: Defining constructors as functions with the same name as the contract is deprecated. Use "constructor(...) { ... }" instead.\n    function Foo() public {}\n    ^----------------------^\n')

FOO_SOURCE = 'pragma solidity ^0.4.17;\n\ncontract Foo {\n    function Foo() public {}\n\n    function return13() public pure returns (uint) {\n        return 13;\n    }\n}\n'
_          = <subprocess.Popen object at 0x7f1aa8150390>
err        = '<stdin>:4:5: Warning: Defining constructors as functions with the same name as the contract is deprecated. Use "constructor(...) { ... }" instead.\n    function Foo() public {}\n    ^----------------------^\n'
output     = '\n======= <stdin>:Foo =======\nBinary: \n6080604052348015600f57600080fd5b5060a18061001e6000396000f3006080604052600436...80910390f35b6000600d9050905600a165627a7a72305820222a1dc8029613612a0659d720a86492832454b7f50a2f689ce684cd93fa671b0029\n'

tests/core/wrapper/test_solc_wrapper.py:38: AssertionError
======================================================================= 1 failed, 57 deselected in 0.07 seconds ========================================================================