From 52b4da1f8e3a10a14a997c52796cb93ee0127977 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Mon, 29 Jan 2018 09:18:23 +0100 Subject: [PATCH 1/5] bpo-31368: Enhance os.preadv() documentation --- Doc/library/os.rst | 110 ++++++++++++++++++++++++++----------------- Doc/whatsnew/3.7.rst | 3 ++ 2 files changed, 69 insertions(+), 44 deletions(-) diff --git a/Doc/library/os.rst b/Doc/library/os.rst index bae432d33b0bd4..e625fdcabd95ec 100644 --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -1104,39 +1104,48 @@ or `the MSDN `_ on Windo .. function:: pwritev(fd, buffers, offset, flags=0) - Combines the functionality of :func:`os.writev` and :func:`os.pwrite`. It - writes the contents of *buffers* to file descriptor *fd* at offset *offset*. - *buffers* must be a sequence of :term:`bytes-like objects `. - Buffers are processed in array order. Entire contents of first buffer is written - before proceeding to second, and so on. The operating system may set a limit - (sysconf() value SC_IOV_MAX) on the number of buffers that can be used. - :func:`~os.pwritev` writes the contents of each object to the file descriptor - and returns the total number of bytes written. + Write the *buffers* contents to file descriptor *fd* at offset *offset*. + *buffers* must be a sequence of :term:`bytes-like objects `. Buffers are processed in array order. Entire contents of first + buffer is written before proceeding to second, and so on. + + Return the total number of bytes written. + + Combine the functionality of :func:`os.writev` and :func:`os.pwrite`. The *flags* argument contains a bitwise OR of zero or more of the following flags: - - RWF_DSYNC - - RWF_SYNC + - :data:`RWF_DSYNC` + - :data:`RWF_SYNC` Using non-zero flags requires Linux 4.7 or newer. - Availability: Linux (version 2.6.30), FreeBSD 6.0 and newer, - OpenBSD (version 2.7 and newer). + The operating system may set a limit (:func:`sysconf` value + ``'SC_IOV_MAX'``) on the number of buffers that can be used. + + Availability: Linux 2.6.30 and newer, FreeBSD 6.0 and newer, + OpenBSD 2.7 and newer. .. versionadded:: 3.7 -.. data:: RWF_DSYNC (since Linux 4.7) - Provide a per-write equivalent of the O_DSYNC open(2) flag. This flag - is meaningful only for pwritev2(), and its effect applies only to the - data range written by the system call. + +.. data:: RWF_DSYNC + + Provide a per-write equivalent of the :data:`O_DSYNC` ``open(2)`` flag. This + flag effect applies only to the data range written by the system call. + + Availability: Linux 4.7 and newer. .. versionadded:: 3.7 -.. data:: RWF_SYNC (since Linux 4.7) - Provide a per-write equivalent of the O_SYNC open(2) flag. This flag is - meaningful only for pwritev2(), and its effect applies only to the data - range written by the system call. + +.. data:: RWF_SYNC + + Provide a per-write equivalent of the :data:`O_SYNC` ``open(2)`` flag. This + flag effect applies only to the data range written by the system call. + + Availability: Linux 4.7 and newer. .. versionadded:: 3.7 @@ -1237,45 +1246,58 @@ or `the MSDN `_ on Windo .. function:: preadv(fd, buffers, offset, flags=0) - Combines the functionality of :func:`os.readv` and :func:`os.pread`. It - reads from a file descriptor *fd* into a number of mutable :term:`bytes-like - objects ` *buffers*. As :func:`os.readv`, it will transfer - data into each buffer until it is full and then move on to the next buffer in - the sequence to hold the rest of the data. Its fourth argument, *offset*, - specifies the file offset at which the input operation is to be performed. - :func:`~os.preadv` return the total number of bytes read (which can be less than - the total capacity of all the objects). + Read from a file descriptor *fd* from offset *offset* into mutable + :term:`bytes-like objects ` *buffers*. Transfer data into + each buffer until it is full and then move on to the next buffer in the + sequence to hold the rest of the data. + + Return the total number of bytes read which can be less than the total + capacity of all the objects. + + Combine the functionality of :func:`os.readv` and :func:`os.pread`. The flags argument contains a bitwise OR of zero or more of the following flags: - - RWF_HIPRI - - RWF_NOWAIT + - :data:`RWF_HIPRI` + - :data:`RWF_NOWAIT` Using non-zero flags requires Linux 4.6 or newer. - Availability: Linux (version 2.6.30), FreeBSD 6.0 and newer, - OpenBSD (version 2.7 and newer). + The operating system may set a limit (:func:`sysconf` value + ``'SC_IOV_MAX'``) on the number of buffers that can be used. + + Availability: Linux 2.6.30 abd newer, FreeBSD 6.0 and newer, + OpenBSD 2.7 and newer. .. versionadded:: 3.7 -.. data:: RWF_HIPRI (since Linux 4.6) - High priority read/write. Allows block-based filesystems to use polling - of the device, which provides lower latency, but may use additional - resources. (Currently, this feature is usable only on a file descriptor - opened using the O_DIRECT flag.) +.. data:: RWF_NOWAIT + + Do not wait for data which is not immediately available. If this flag is + specified, the system call will return instantly if it would have to read + data from the backing storage or wait for a lock. + + If some data was successfully read, it will return the number of bytes read. + If no bytes were read, it will return ``-1`` and set errno to + :data:`errno.EAGAIN`. + + Availability: Linux 4.14 and newer. .. versionadded:: 3.7 -.. data:: RWF_NOWAIT (since Linux 4.14) - Do not wait for data which is not immediately available. If this flag - is specified, the preadv2() system call will return instantly - if it would have to read data from the backing storage or wait for a lock. - If some data was successfully read, it will return the number of bytes - read. If no bytes were read, it will return -1 and set errno to EAGAIN. - Currently, this flag is meaningful only for preadv2(). +.. data:: RWF_HIPRI + + High priority read/write. Allows block-based filesystems to use polling + of the device, which provides lower latency, but may use additional + resources. + + Currently, on Linux, this feature is usable only on a file descriptor opened + using the O_DIRECT flag. + + Availability: Linux 4.6 and newer. .. versionadded:: 3.7 diff --git a/Doc/whatsnew/3.7.rst b/Doc/whatsnew/3.7.rst index 9979e69239907a..32a28df0232b4f 100644 --- a/Doc/whatsnew/3.7.rst +++ b/Doc/whatsnew/3.7.rst @@ -554,6 +554,9 @@ operation. (Contributed by Mark Dickinson in :issue:`29962`.) os -- +Added :func:`os.preadv` and :func:`os.pwritev` functions. (Contributed by Pablo +Galindo in :issue:`31368`.) + Added support for :class:`bytes` paths in :func:`~os.fwalk`. (Contributed by Serhiy Storchaka in :issue:`28682`.) From 5809a57f1e622a4fc4f01924ec26290d8342106f Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Mon, 29 Jan 2018 09:23:18 +0100 Subject: [PATCH 2/5] What's New: describe added functions --- Doc/whatsnew/3.7.rst | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/Doc/whatsnew/3.7.rst b/Doc/whatsnew/3.7.rst index 32a28df0232b4f..28a6a1ba41a9ef 100644 --- a/Doc/whatsnew/3.7.rst +++ b/Doc/whatsnew/3.7.rst @@ -554,8 +554,10 @@ operation. (Contributed by Mark Dickinson in :issue:`29962`.) os -- -Added :func:`os.preadv` and :func:`os.pwritev` functions. (Contributed by Pablo -Galindo in :issue:`31368`.) +Added :func:`os.preadv` (combine the functionality of :func:`os.readv` and +:func:`os.pread`) and :func:`os.pwritev` functions (combine the functionality +of :func:`os.writev` and :func:`os.pwrite`). (Contributed by Pablo Galindo in +:issue:`31368`.) Added support for :class:`bytes` paths in :func:`~os.fwalk`. (Contributed by Serhiy Storchaka in :issue:`28682`.) From 14eafd21868c554476cb0949a2270542f0da2bf4 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Mon, 29 Jan 2018 16:37:17 +0100 Subject: [PATCH 3/5] Adding missing "the" in the doc --- Doc/library/os.rst | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/Doc/library/os.rst b/Doc/library/os.rst index e625fdcabd95ec..8fe88c2839a357 100644 --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -1106,8 +1106,8 @@ or `the MSDN `_ on Windo Write the *buffers* contents to file descriptor *fd* at offset *offset*. *buffers* must be a sequence of :term:`bytes-like objects `. Buffers are processed in array order. Entire contents of first - buffer is written before proceeding to second, and so on. + object>`. Buffers are processed in array order. Entire contents of the first + buffer is written before proceeding to the second, and so on. Return the total number of bytes written. @@ -1350,9 +1350,10 @@ or `the MSDN `_ on Windo Write the contents of *buffers* to file descriptor *fd*. *buffers* must be a sequence of :term:`bytes-like objects `. Buffers are - processed in array order. Entire contents of first buffer is written before - proceeding to second, and so on. The operating system may set a limit - (sysconf() value SC_IOV_MAX) on the number of buffers that can be used. + processed in array order. Entire contents of the first buffer is written + before proceeding to the second, and so on. The operating system may set a + limit (sysconf() value SC_IOV_MAX) on the number of buffers that can be + used. :func:`~os.writev` writes the contents of each object to the file descriptor and returns the total number of bytes written. From 622a8ec053f54f933d57bb8e8346f8e92e56890f Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Mon, 29 Jan 2018 17:01:19 +0100 Subject: [PATCH 4/5] Move preadv to fix ordering --- Doc/library/os.rst | 116 ++++++++++++++++++++++----------------------- 1 file changed, 58 insertions(+), 58 deletions(-) diff --git a/Doc/library/os.rst b/Doc/library/os.rst index 8fe88c2839a357..15a13f700f6709 100644 --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -1092,6 +1092,64 @@ or `the MSDN `_ on Windo .. versionadded:: 3.3 +.. function:: preadv(fd, buffers, offset, flags=0) + + Read from a file descriptor *fd* from offset *offset* into mutable + :term:`bytes-like objects ` *buffers*. Transfer data into + each buffer until it is full and then move on to the next buffer in the + sequence to hold the rest of the data. + + Return the total number of bytes read which can be less than the total + capacity of all the objects. + + Combine the functionality of :func:`os.readv` and :func:`os.pread`. + + The flags argument contains a bitwise OR of zero or more of the following + flags: + + - :data:`RWF_HIPRI` + - :data:`RWF_NOWAIT` + + Using non-zero flags requires Linux 4.6 or newer. + + The operating system may set a limit (:func:`sysconf` value + ``'SC_IOV_MAX'``) on the number of buffers that can be used. + + Availability: Linux 2.6.30 abd newer, FreeBSD 6.0 and newer, + OpenBSD 2.7 and newer. + + .. versionadded:: 3.7 + + +.. data:: RWF_NOWAIT + + Do not wait for data which is not immediately available. If this flag is + specified, the system call will return instantly if it would have to read + data from the backing storage or wait for a lock. + + If some data was successfully read, it will return the number of bytes read. + If no bytes were read, it will return ``-1`` and set errno to + :data:`errno.EAGAIN`. + + Availability: Linux 4.14 and newer. + + .. versionadded:: 3.7 + + +.. data:: RWF_HIPRI + + High priority read/write. Allows block-based filesystems to use polling + of the device, which provides lower latency, but may use additional + resources. + + Currently, on Linux, this feature is usable only on a file descriptor opened + using the O_DIRECT flag. + + Availability: Linux 4.6 and newer. + + .. versionadded:: 3.7 + + .. function:: pwrite(fd, str, offset) Write *bytestring* to a file descriptor, *fd*, from *offset*, @@ -1244,64 +1302,6 @@ or `the MSDN `_ on Windo .. versionadded:: 3.3 -.. function:: preadv(fd, buffers, offset, flags=0) - - Read from a file descriptor *fd* from offset *offset* into mutable - :term:`bytes-like objects ` *buffers*. Transfer data into - each buffer until it is full and then move on to the next buffer in the - sequence to hold the rest of the data. - - Return the total number of bytes read which can be less than the total - capacity of all the objects. - - Combine the functionality of :func:`os.readv` and :func:`os.pread`. - - The flags argument contains a bitwise OR of zero or more of the following - flags: - - - :data:`RWF_HIPRI` - - :data:`RWF_NOWAIT` - - Using non-zero flags requires Linux 4.6 or newer. - - The operating system may set a limit (:func:`sysconf` value - ``'SC_IOV_MAX'``) on the number of buffers that can be used. - - Availability: Linux 2.6.30 abd newer, FreeBSD 6.0 and newer, - OpenBSD 2.7 and newer. - - .. versionadded:: 3.7 - - -.. data:: RWF_NOWAIT - - Do not wait for data which is not immediately available. If this flag is - specified, the system call will return instantly if it would have to read - data from the backing storage or wait for a lock. - - If some data was successfully read, it will return the number of bytes read. - If no bytes were read, it will return ``-1`` and set errno to - :data:`errno.EAGAIN`. - - Availability: Linux 4.14 and newer. - - .. versionadded:: 3.7 - - -.. data:: RWF_HIPRI - - High priority read/write. Allows block-based filesystems to use polling - of the device, which provides lower latency, but may use additional - resources. - - Currently, on Linux, this feature is usable only on a file descriptor opened - using the O_DIRECT flag. - - Availability: Linux 4.6 and newer. - - .. versionadded:: 3.7 - - .. function:: tcgetpgrp(fd) Return the process group associated with the terminal given by *fd* (an open From 40457cba2a601d418aee52c2f2d225065a272784 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Mon, 29 Jan 2018 17:58:32 +0100 Subject: [PATCH 5/5] Cleanup other read/write docs pread(): rename second parameter from 'buffersize' to 'n', reuse read() parameter name. The parameter is positional-only and so this change is backward compatible. --- Doc/library/os.rst | 96 +++++++++++++++++++++++++--------------------- 1 file changed, 52 insertions(+), 44 deletions(-) diff --git a/Doc/library/os.rst b/Doc/library/os.rst index 15a13f700f6709..fd0bd5df4c53d8 100644 --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -1082,10 +1082,13 @@ or `the MSDN `_ on Windo .. versionadded:: 3.3 -.. function:: pread(fd, buffersize, offset) +.. function:: pread(fd, n, offset) - Read from a file descriptor, *fd*, at a position of *offset*. It will read up - to *buffersize* number of bytes. The file offset remains unchanged. + Read at most *n* bytes from file descriptor *fd* at a position of *offset*, + leaving the file offset unchanged. + + Return a bytestring containing the bytes read. If the end of the file + referred to by *fd* has been reached, an empty bytes object is returned. Availability: Unix. @@ -1094,15 +1097,10 @@ or `the MSDN `_ on Windo .. function:: preadv(fd, buffers, offset, flags=0) - Read from a file descriptor *fd* from offset *offset* into mutable - :term:`bytes-like objects ` *buffers*. Transfer data into - each buffer until it is full and then move on to the next buffer in the - sequence to hold the rest of the data. - - Return the total number of bytes read which can be less than the total - capacity of all the objects. - - Combine the functionality of :func:`os.readv` and :func:`os.pread`. + Read from a file descriptor *fd* at a position of *offset* into mutable + :term:`bytes-like objects ` *buffers*, leaving the file + offset unchanged. Transfer data into each buffer until it is full and then + move on to the next buffer in the sequence to hold the rest of the data. The flags argument contains a bitwise OR of zero or more of the following flags: @@ -1110,13 +1108,16 @@ or `the MSDN `_ on Windo - :data:`RWF_HIPRI` - :data:`RWF_NOWAIT` - Using non-zero flags requires Linux 4.6 or newer. + Return the total number of bytes actually read which can be less than the + total capacity of all the objects. The operating system may set a limit (:func:`sysconf` value ``'SC_IOV_MAX'``) on the number of buffers that can be used. - Availability: Linux 2.6.30 abd newer, FreeBSD 6.0 and newer, - OpenBSD 2.7 and newer. + Combine the functionality of :func:`os.readv` and :func:`os.pread`. + + Availability: Linux 2.6.30 and newer, FreeBSD 6.0 and newer, + OpenBSD 2.7 and newer. Using flags requires Linux 4.6 or newer. .. versionadded:: 3.7 @@ -1152,8 +1153,10 @@ or `the MSDN `_ on Windo .. function:: pwrite(fd, str, offset) - Write *bytestring* to a file descriptor, *fd*, from *offset*, - leaving the file offset unchanged. + Write the bytestring in *str* to file descriptor *fd* at position of + *offset*, leaving the file offset unchanged. + + Return the number of bytes actually written. Availability: Unix. @@ -1162,14 +1165,11 @@ or `the MSDN `_ on Windo .. function:: pwritev(fd, buffers, offset, flags=0) - Write the *buffers* contents to file descriptor *fd* at offset *offset*. - *buffers* must be a sequence of :term:`bytes-like objects `. Buffers are processed in array order. Entire contents of the first - buffer is written before proceeding to the second, and so on. - - Return the total number of bytes written. - - Combine the functionality of :func:`os.writev` and :func:`os.pwrite`. + Write the *buffers* contents to file descriptor *fd* at a offset *offset*, + leaving the file offset unchanged. *buffers* must be a sequence of + :term:`bytes-like objects `. Buffers are processed in + array order. Entire contents of the first buffer is written before + proceeding to the second, and so on. The *flags* argument contains a bitwise OR of zero or more of the following flags: @@ -1177,13 +1177,15 @@ or `the MSDN `_ on Windo - :data:`RWF_DSYNC` - :data:`RWF_SYNC` - Using non-zero flags requires Linux 4.7 or newer. + Return the total number of bytes actually written. The operating system may set a limit (:func:`sysconf` value ``'SC_IOV_MAX'``) on the number of buffers that can be used. + Combine the functionality of :func:`os.writev` and :func:`os.pwrite`. + Availability: Linux 2.6.30 and newer, FreeBSD 6.0 and newer, - OpenBSD 2.7 and newer. + OpenBSD 2.7 and newer. Using flags requires Linux 4.7 or newer. .. versionadded:: 3.7 @@ -1210,9 +1212,10 @@ or `the MSDN `_ on Windo .. function:: read(fd, n) - Read at most *n* bytes from file descriptor *fd*. Return a bytestring containing the - bytes read. If the end of the file referred to by *fd* has been reached, an - empty bytes object is returned. + Read at most *n* bytes from file descriptor *fd*. + + Return a bytestring containing the bytes read. If the end of the file + referred to by *fd* has been reached, an empty bytes object is returned. .. note:: @@ -1291,11 +1294,15 @@ or `the MSDN `_ on Windo .. function:: readv(fd, buffers) Read from a file descriptor *fd* into a number of mutable :term:`bytes-like - objects ` *buffers*. :func:`~os.readv` will transfer data - into each buffer until it is full and then move on to the next buffer in the - sequence to hold the rest of the data. :func:`~os.readv` returns the total - number of bytes read (which may be less than the total capacity of all the - objects). + objects ` *buffers*. Transfer data into each buffer until + it is full and then move on to the next buffer in the sequence to hold the + rest of the data. + + Return the total number of bytes actually read which can be less than the + total capacity of all the objects. + + The operating system may set a limit (:func:`sysconf` value + ``'SC_IOV_MAX'``) on the number of buffers that can be used. Availability: Unix. @@ -1329,8 +1336,9 @@ or `the MSDN `_ on Windo .. function:: write(fd, str) - Write the bytestring in *str* to file descriptor *fd*. Return the number of - bytes actually written. + Write the bytestring in *str* to file descriptor *fd*. + + Return the number of bytes actually written. .. note:: @@ -1348,15 +1356,15 @@ or `the MSDN `_ on Windo .. function:: writev(fd, buffers) - Write the contents of *buffers* to file descriptor *fd*. *buffers* must be a - sequence of :term:`bytes-like objects `. Buffers are + Write the contents of *buffers* to file descriptor *fd*. *buffers* must be + a sequence of :term:`bytes-like objects `. Buffers are processed in array order. Entire contents of the first buffer is written - before proceeding to the second, and so on. The operating system may set a - limit (sysconf() value SC_IOV_MAX) on the number of buffers that can be - used. + before proceeding to the second, and so on. - :func:`~os.writev` writes the contents of each object to the file descriptor - and returns the total number of bytes written. + Returns the total number of bytes actually written. + + The operating system may set a limit (:func:`sysconf` value + ``'SC_IOV_MAX'``) on the number of buffers that can be used. Availability: Unix.