➜

This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Update to expat 2.2.4 (expat: utf8_toUtf8 cannot properly handle exhausting buffer)
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.4, Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: benjamin.peterson, georg.brandl, larry, ned.deily, serhiy.storchaka, sping, tianlynn, vstinner
Priority: release blocker Keywords: patch

Created on 2017-08-10 04:48 by tianlynn, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
cpython_rebuild_expat_dir.sh vstinner, 2017-09-04 21:41
Pull Requests
URL Status Linked Edit
PR 3315 merged vstinner, 2017-09-04 21:40
PR 3350 merged vstinner, 2017-09-05 18:29
PR 3351 closed vstinner, 2017-09-05 18:30
PR 3352 merged vstinner, 2017-09-05 18:37
PR 3353 merged vstinner, 2017-09-05 18:41
PR 3354 merged vstinner, 2017-09-05 18:44
PR 3570 merged vstinner, 2017-09-14 09:27
PR 3745 merged vstinner, 2017-09-25 08:31
PR 3746 merged vstinner, 2017-09-25 08:37
PR 3751 merged steve.dower, 2017-09-25 16:46
PR 3785 merged vstinner, 2017-09-27 08:51
Messages (26)
msg300043 - (view) Author: Lin Tian (tianlynn) * Date: 2017-08-10 04:48 
utf8_toUtf8(const ENCODING *UNUSED_P(enc),
            const char **fromP, const char *fromLim,
            char **toP, const char *toLim)
{
  char *to;
  const char *from;
  const char *fromLimInitial = fromLim;

  /* Avoid copying partial characters. */
  align_limit_to_full_utf8_characters(*fromP, &fromLim);

  for (to = *toP, from = *fromP; (from < fromLim) && (to < toLim); from++, to++)
    *to = *from;
  *fromP = from;
  *toP = to;

  if (fromLim < fromLimInitial)
    return XML_CONVERT_INPUT_INCOMPLETE;
  else if ((to == toLim) && (from < fromLim))
    // <===== Bug is here. In case (to == toLim), it's possible that
    //        from is still pointing to partial character. For example,
    //        a character with 3 bytes (A, B, C) and form is pointing to C.
    //        It means only A and B is copied to output buffer. Next
    //        scanning will start with C which could be considered as invalid
    //        byte and got dropped. After this, only "AB" is kept in memory
    //        and thus it will lead to invalid continuation byte.
    return XML_CONVERT_OUTPUT_EXHAUSTED;
  else
    return XML_CONVERT_COMPLETED;
}
msg300044 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-08-10 04:58 
This is not our code. Please use corresponding bug tracker [1] if you have found a bug in Expat.

But I think there is not a bug here.

[1] https://github.com/libexpat/libexpat/issues
msg300268 - (view) Author: Lin Tian (tianlynn) * Date: 2017-08-14 22:44 
Reactive this issue as to let you know that libexpat has confirmed and fixed the bug and they are interested in porting the fix to python. Reactive this in case you want to know what's going on and make a decision accordingly. (Sorry, I'm not very familiar with process here)
msg300309 - (view) Author: (sping) * Date: 2017-08-15 18:38 
For the record, the upstream fix is commit https://github.com/libexpat/libexpat/commit/74a7090a6eb92c27b7010287a4082de6b357fa42 and it will be part of Expat 2.2.4.
msg300605 - (view) Author: (sping) * Date: 2017-08-20 18:28 
Expat 2.2.4 with a fix has been released now: https://github.com/libexpat/libexpat/releases
msg301011 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-08-30 05:25 
I confirm the regression (see issue31303 for reproducer). Victor, do you mind to update expat to 2.2.4?

This issue can be classified as a security issue, since a regression was added in security update.
msg301012 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-08-30 05:41 
Corresponding Expat issue: https://github.com/libexpat/libexpat/issues/115.
msg301084 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2017-09-01 04:58 
We can put expat 2.2.4 in 2.7.14 final.
msg301270 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-09-04 21:42 
I produced attached PR 3315 using attached cpython_rebuild_expat_dir.sh + revert Modules/expat/expat_external.h change to keep #include "pyexpatns.h".
msg301287 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-09-04 23:58 
New changeset 759e30ec47048cb9835c62aaeac48748c8151390 by Victor Stinner in branch 'master':
bpo-31170: Update libexpat from 2.2.3 to 2.2.4 (#3315)
https://github.com/python/cpython/commit/759e30ec47048cb9835c62aaeac48748c8151390
msg301399 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-09-05 23:30 
New changeset e5f2f8038540f9f06478f842f8f7313df4d2e59b by Victor Stinner in branch '3.6':
bpo-31170: Update libexpat from 2.2.3 to 2.2.4 (#3315) (#3350)
https://github.com/python/cpython/commit/e5f2f8038540f9f06478f842f8f7313df4d2e59b
msg301422 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2017-09-06 00:57 
New changeset 297516ea509c72d8ebed3a9b3ce200f023aca0b7 by Ned Deily (Victor Stinner) in branch '3.3':
[3.3] bpo-30947, bpo-31170: Update expat from 2.2.1 to 2.2.4 (#3352)
https://github.com/python/cpython/commit/297516ea509c72d8ebed3a9b3ce200f023aca0b7
msg301424 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2017-09-06 01:07 
New changeset c00d5087cd308cc8be41e0afd8df27726185347f by Benjamin Peterson (Victor Stinner) in branch '2.7':
bpo-31170: Update libexpat from 2.2.3 to 2.2.4 (#3315)
https://github.com/python/cpython/commit/c00d5087cd308cc8be41e0afd8df27726185347f
msg302159 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-09-14 09:30 
I wrote an non-regression test for the Python master branch using the test case attached to https://github.com/libexpat/libexpat/issues/115: PR 3570.
msg302483 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2017-09-18 19:43 
I believe all the branches except 3.5 and 3.4 have been updated so adjusting the Versions field accordingly.  All yours, Larry!
msg302833 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2017-09-24 08:04 
New changeset 86a713cb0c110b6798ca7f9e630fc511ee0a4028 by larryhastings (Victor Stinner) in branch '3.4':
[3.4][Security] bpo-30947, bpo-31170: Update expat from 2.2.1 to 2.2.4 (#3353)
https://github.com/python/cpython/commit/86a713cb0c110b6798ca7f9e630fc511ee0a4028
msg302898 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2017-09-25 00:58 
New changeset f2492bb6aae061aea47e21fc7e56b7ab9bfdf543 by larryhastings (Victor Stinner) in branch '3.5':
[3.5][Security] bpo-30947, bpo-31170: Update expat from 2.2.1 to 2.2.4 (#3354)
https://github.com/python/cpython/commit/f2492bb6aae061aea47e21fc7e56b7ab9bfdf543
msg302925 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-09-25 08:27 
New changeset e6d9fcbb8d0c325e57df08ae8781aafedb71eca2 by Victor Stinner in branch 'master':
bpo-31170: Write unit test for Expat 2.2.4 UTF-8 bug (#3570)
https://github.com/python/cpython/commit/e6d9fcbb8d0c325e57df08ae8781aafedb71eca2
msg302927 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-09-25 08:43 
New changeset 5f5da728aec9c4f74cc771fbf30037b64a447514 by Victor Stinner in branch '2.7':
bpo-31170: Write unit test for Expat 2.2.4 UTF-8 bug (#3570) (#3745)
https://github.com/python/cpython/commit/5f5da728aec9c4f74cc771fbf30037b64a447514
msg302930 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-09-25 09:00 
New changeset ad051cbce1360ad3055a048506c09bc2a5442474 by Victor Stinner in branch '3.6':
bpo-31170: Write unit test for Expat 2.2.4 UTF-8 bug (#3570) (#3746)
https://github.com/python/cpython/commit/ad051cbce1360ad3055a048506c09bc2a5442474
msg302931 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-09-25 09:02 
libexpat has been upgraded to version 2.2.4 in Python 2.7, 3.3, 3.4, 3.5, 3.6 and master.

I added an unit test to Python 2.7, 3.6 and master. I'm not sure about backporting the new unit test to Python 3.3, 3.4 and 3.5.

I close the issue.
msg302975 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-09-25 19:48 
Oh, it seems like Steve Dower found an issue on Windows: PR 3751.

I reopen the issue.
msg303037 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2017-09-26 13:00 
New changeset 44c1b62939a6192776dc9d093546154044cb2ecb by larryhastings (Steve Dower) in branch '3.5':
[3.5] bpo-31170: Fix inclusion of expat in Windows build projects. (#3751)
https://github.com/python/cpython/commit/44c1b62939a6192776dc9d093546154044cb2ecb
msg307254 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2017-11-29 18:50 
New changeset 8b11e8de7aedacfbbcc8c780f3c4097396f1d1a3 by larryhastings (Victor Stinner) in branch '3.4':
[3.4] bpo-31170: Fix inclusion of expat in Windows build projects (#3785)
https://github.com/python/cpython/commit/8b11e8de7aedacfbbcc8c780f3c4097396f1d1a3
msg309469 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-01-04 14:45 
Can this issue be closed now?
msg309471 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-01-04 15:52 
Ah yes, it can be closed. I was waiting 3.4 and 3.5 fixes to be fixed, which is now the case.
History
Date User Action Args
2022-04-11 14:58:49adminsetgithub: 75353
2019-05-10 18:04:35ned.deilysetmessages: - msg342109
2019-05-10 17:36:40ned.deilysetmessages: + msg342109
2018-01-04 15:52:09vstinnersetstatus: pending -> closed
resolution: fixed
messages: + msg309471

stage: patch review -> resolved
2018-01-04 14:45:34serhiy.storchakasetstatus: open -> pending

messages: + msg309469
2017-11-29 18:50:44larrysetmessages: + msg307254
2017-09-27 08:51:11vstinnersetstage: resolved -> patch review
pull_requests: + pull_request3770
2017-09-26 13:00:30larrysetmessages: + msg303037
2017-09-25 19:48:47vstinnersetstatus: closed -> open
resolution: fixed -> (no value)
messages: + msg302975
2017-09-25 16:46:06steve.dowersetpull_requests: + pull_request3738
2017-09-25 09:02:31vstinnersetstatus: open -> closed
resolution: third party -> fixed
messages: + msg302931

stage: patch review -> resolved
2017-09-25 09:00:06vstinnersetmessages: + msg302930
2017-09-25 08:43:58vstinnersetmessages: + msg302927
2017-09-25 08:37:18vstinnersetpull_requests: + pull_request3733
2017-09-25 08:31:43vstinnersetstage: commit review -> patch review
pull_requests: + pull_request3732
2017-09-25 08:27:37vstinnersetmessages: + msg302925
2017-09-25 00:58:35larrysetmessages: + msg302898
2017-09-24 08:04:56larrysetmessages: + msg302833
2017-09-18 19:43:07ned.deilysetstage: patch review -> commit review
messages: + msg302483
versions: - Python 3.3, Python 3.6, Python 3.7
2017-09-16 17:35:29benjamin.petersonsetversions: - Python 2.7
2017-09-14 09:30:15vstinnersetmessages: + msg302159
2017-09-14 09:27:19vstinnersetkeywords: + patch
stage: resolved -> patch review
pull_requests: + pull_request3559
2017-09-06 01:07:08benjamin.petersonsetmessages: + msg301424
2017-09-06 00:57:39ned.deilysetmessages: + msg301422
2017-09-05 23:30:24vstinnersetmessages: + msg301399
2017-09-05 18:44:44vstinnersetpull_requests: + pull_request3369
2017-09-05 18:41:15vstinnersetpull_requests: + pull_request3367
2017-09-05 18:37:37vstinnersetpull_requests: + pull_request3365
2017-09-05 18:30:54vstinnersetpull_requests: + pull_request3363
2017-09-05 18:29:39vstinnersetpull_requests: + pull_request3362
2017-09-04 23:58:13vstinnersetmessages: + msg301287
2017-09-04 21:42:52vstinnersetmessages: + msg301270
2017-09-04 21:41:01vstinnersetfiles: + cpython_rebuild_expat_dir.sh
2017-09-04 21:40:47vstinnersetpull_requests: + pull_request3342
2017-09-04 21:36:37vstinnersettitle: expat: utf8_toUtf8 cannot properly handle exhausting buffer -> Update to expat 2.2.4 (expat: utf8_toUtf8 cannot properly handle exhausting buffer)
2017-09-01 04:58:19benjamin.petersonsetmessages: + msg301084
2017-08-30 05:41:03serhiy.storchakasetmessages: + msg301012
2017-08-30 05:25:44serhiy.storchakasetpriority: normal -> release blocker
nosy: + ned.deily, benjamin.peterson, georg.brandl, larry
messages: + msg301011

2017-08-30 05:15:12serhiy.storchakalinkissue31303 superseder
2017-08-21 05:10:05serhiy.storchakasetnosy: + vstinner
2017-08-20 18:28:40spingsetmessages: + msg300605
2017-08-15 18:38:42spingsetnosy: + sping

messages: + msg300309
versions: + Python 2.7, Python 3.3, Python 3.4, Python 3.5
2017-08-14 22:44:19tianlynnsetstatus: closed -> open

messages: + msg300268
2017-08-10 04:59:02serhiy.storchakasetstatus: open -> closed
2017-08-10 04:58:29serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg300044

resolution: third party
stage: resolved
2017-08-10 04:48:25tianlynncreate