AMD64 Windows8.1 Refleaks 3.x:
http://buildbot.python.org/all/#/builders/80/builds/118

test_asyncio leaked [4, 4, 3] memory blocks, sum=11

I reproduced the issue. I'm running test.bisect to try to isolate this bug.

It seems to be related to sendfile():

C:\vstinner\python\master>python -m test -R 3:3 test_asyncio -m test.test_asyncio.test_events.ProactorEventLoopTests.test_sendfile_close_peer_in_middle_of_receiving
Running Debug|x64 interpreter...
Run tests sequentially
0:00:00 [1/1] test_asyncio
beginning 6 repetitions
123456
......
test_asyncio leaked [1, 2, 1] memory blocks, sum=4
test_asyncio failed

1 test failed:
    test_asyncio

Total duration: 1 sec
Tests result: FAILURE

Andrew, please take a look.  I'll have very limited (to no at all) time to work on Python in the next 2 weeks.

The test is still leaking memory blocks. Any progress on investigating the issue?

Oh, I found again this bug while working on my PR 7827 (detect handle leaks on Windows in regrtest).

See also bpo-33735: my commit 23401fb960bb94e6ea62d2999527968d53d3fc65 fixes a false alarm in regrtest when hunting leaks in test_multiprocessing_spawn or test_multiprocessing_forkserver.

New changeset df8e1fb4e388e18430a9be8c6ceeb03c330f166c by Victor Stinner in branch 'master':
bpo-32710: test_asyncio: test_sendfile reset policy (GH-11461)
https://github.com/python/cpython/commit/df8e1fb4e388e18430a9be8c6ceeb03c330f166c

Update:

* test.test_asyncio.test_sendfile.ProactorEventLoopTests.test_sendfile_close_peer_in_the_middle_of_receiving leaks 1 reference per run: this bug is caused by bpo-35682 and fixed by PR 11462
* test.test_asyncio.test_sendfile.ProactorEventLoopTests.test_sendfile_fallback_close_peer_in_the_middle_of_receiving leaks 1 reference per run: I don't understand why.

I spent a lot of time to investigate test_sendfile_fallback_close_peer_in_the_middle_of_receiving() leak and I don't understand the issue. The main loop is BaseEventLoop._sendfile_fallback(). For the specific case of this test, the loop can be simplified to:

        proto = _SendfileFallbackProtocol(transp)
        try:
            while True:
                data = b'x' * (1024 * 64)
                await proto.drain()
                transp.write(data)
        finally:
            await proto.restore()

The server closes the connection after it gets 1024 bytes. The client socket gets a ConnectionAbortedError exception in _ProactorBaseWritePipeTransport._loop_writing() which calls _fatal_error():

        except OSError as exc:
            self._fatal_error(exc, 'Fatal write error on pipe transport')

_fatal_error() calls _force_close() which sets _closing to True and calls protocol.connection_lost(). In the meanwhile, drain() raises ConnectionError because is_closing() is true:

    async def drain(self):
        if self._transport.is_closing():
            raise ConnectionError("Connection closed by peer")
        ...

Interesting commit. No idea if it's related.

commit 79790bc35fe722a49977b52647f9b5fe1deda2b7
Author: Victor Stinner <vstinner@redhat.com>
Date:   Fri Jun 8 00:25:52 2018 +0200

    bpo-33694: Fix race condition in asyncio proactor (GH-7498)
    
    The cancellation of an overlapped WSARecv() has a race condition
    which causes data loss because of the current implementation of
    proactor in asyncio.
    
    No longer cancel overlapped WSARecv() in _ProactorReadPipeTransport
    to work around the race condition.
    
    Remove the optimized recv_into() implementation to get simple
    implementation of pause_reading() using the single _pending_data
    attribute.
    
    Move _feed_data_to_bufferred_proto() to protocols.py.
    
    Remove set_protocol() method which became useless.

Attached test_aiosend.py is a simplified version of test to trigger the reference leak. Copy it to Lib/test/ and run:

vstinner@WIN C:\vstinner\python\master>python -m test test_aiosend -R 3:3
Running Debug|x64 interpreter...
Run tests sequentially
0:00:00 [1/1] test_aiosend
beginning 6 repetitions
123456
......
test_aiosend leaked [1, 1, 1] references, sum=3
test_aiosend leaked [1, 2, 1] memory blocks, sum=4
test_aiosend failed

== Tests result: FAILURE ==

1 test failed:
    test_aiosend

Total duration: 548 ms
Tests result: FAILURE

It took me 1 year, a few sleepless nights, multiple attempts to understand the leak, but I eventually found it! WSASend() doesn't release the memory if it fails immediately. I wrote PR 11469 to fix the memory leak. ReadFile() has the same bug, I also fixed it.

By the way, the _overlapped.Overlapped type has no traverse function: it may help the garbage collector to add once, since asyncio is famous for building reference cycles by design (Future.set_exception()).

New changeset a234e148394c2c7419372ab65b773d53a57f3625 by Victor Stinner in branch 'master':
bpo-32710: Fix leak in Overlapped_WSASend() (GH-11469)
https://github.com/python/cpython/commit/a234e148394c2c7419372ab65b773d53a57f3625

I ran test_asyncio refleak hunting on Windows, and there is no more leak!

vstinner@WIN C:\vstinner\python\master>python -m test test_asyncio -R 3:3 
(...)
Total duration: 13 min 24 sec
Tests result: SUCCESS

I will close this PR once the 3.7 backport is merged.

New changeset 88ad48bc98980a40591cc5521703dbb0ad3a9b17 by Miss Islington (bot) in branch '3.7':
bpo-32710: Fix leak in Overlapped_WSASend() (GH-11469)
https://github.com/python/cpython/commit/88ad48bc98980a40591cc5521703dbb0ad3a9b17

New changeset 5485085b324a45307c1ff4ec7d85b5998d7d5e0d by Victor Stinner in branch 'master':
bpo-32710: Fix _overlapped.Overlapped memory leaks (GH-11489)
https://github.com/python/cpython/commit/5485085b324a45307c1ff4ec7d85b5998d7d5e0d

New changeset 059997d78ed1a1a5a364b1846ac972c98c704927 by Miss Islington (bot) in branch '3.7':
bpo-32710: Fix _overlapped.Overlapped memory leaks (GH-11489)
https://github.com/python/cpython/commit/059997d78ed1a1a5a364b1846ac972c98c704927

Ok, _overlapped.Overlapped should now have a few less memory leaks :-)