By "stats" I mean the internal numbers gathered by the VM for performance and debugging. This has nothing to do with any statistics module.

Currently various parts of the VM gather stats: the GC, dicts, the bytecode interpreter, type lookup cache, etc.

These stats have various compile time flags to turn them on and off. They have differing ways to display the stats, and no unified way to gather stats across different runs.

For the specialization stats we dump stats, which we can parse to collate stats across runs.

We should:
1. Add a --with-pystats config flag (like with-pydebug) to turn on stat gathering at build time.
2. Convert the other stats to the format used by the specialization stats, so all stats can be parsed and collated.

New changeset 342b93f9f28746abb7b221a61d5a9b26ccbb395a by Mark Shannon in branch 'main':
bpo-46072: Add --with-pystats configure option to simplify gathering of VM stats (GH-30116)
https://github.com/python/cpython/commit/342b93f9f28746abb7b221a61d5a9b26ccbb395a

Could you please add the new option to Doc/using/configure.rst ?

New changeset 4506bbede1644e985991884964b43afa7ee6f609 by Mark Shannon in branch 'main':
bpo-46072: Document --enable-stats option. (GH-30139)
https://github.com/python/cpython/commit/4506bbede1644e985991884964b43afa7ee6f609

I just noticed that you are using hard-coded paths with /tmp for the pystats directory. That's problematic and opens the possibility of a symlink race attack.

Could please add exclusive create to _Py_PrintSpecializationStats()? The will prevent symlink attacks. fopen() mode "x" is not generally available in all libcs. You have to combine open() and fdopen():


int flags = O_WRONLY | O_CREAT | O_EXCL;
#ifdef O_NOFOLLOW
flags |= O_NOFOLLOW;
#endif
#ifdef O_CLOEXEC
flags |= O_CLOEXEC;
#endif

int fd = open(path, flags);
if (fd >= 0) {
    FILE *fout = fdopen(fd, "w");
}

The --enable-stats option is for CPython development and shouldn't be turned on for a release version, so I'm not really concerned about people attacking their own machines.

New changeset efd6236d36b292c2c43540132c87cf8425e8d627 by Mark Shannon in branch 'main':
bpo-46072:  Add top level stats struct (GH-30169)
https://github.com/python/cpython/commit/efd6236d36b292c2c43540132c87cf8425e8d627

New changeset 90ab138bbdc63763ad825ed6d4821367c09c4015 by Mark Shannon in branch 'main':
bpo-46072: Add simple stats for Python calls. (GH-30989)
https://github.com/python/cpython/commit/90ab138bbdc63763ad825ed6d4821367c09c4015

New changeset 48be46ec1f3f8010570165daa1da4bf9961f3a83 by Mark Shannon in branch 'main':
bpo-46072: Add some object layout and allocation stats (GH-31051)
https://github.com/python/cpython/commit/48be46ec1f3f8010570165daa1da4bf9961f3a83

New changeset 187930f74c44e460ba09c60ba5d9bb4fac543d8f by Mark Shannon in branch 'main':
bpo-46072: Add some frame stats. (GH-31060)
https://github.com/python/cpython/commit/187930f74c44e460ba09c60ba5d9bb4fac543d8f

New changeset 062460e8fd54e53c9a1a6f175ef49c9d730851b8 by Mark Shannon in branch 'main':
bpo-46072: Improve LOAD_METHOD stats (GH-31104)
https://github.com/python/cpython/commit/062460e8fd54e53c9a1a6f175ef49c9d730851b8

New changeset 9c979d7afd839abbb080028bdfeb73727e5cf633 by Mark Shannon in branch 'main':
bpo-46072: Merge dxpairs into py_stats. (GH-31197)
https://github.com/python/cpython/commit/9c979d7afd839abbb080028bdfeb73727e5cf633

New changeset f71a69aa9209cf67cc1060051b147d6afa379bba by Mark Shannon in branch 'main':
bpo-46072: Output stats as markdown with collapsible sections. (GH-31228)
https://github.com/python/cpython/commit/f71a69aa9209cf67cc1060051b147d6afa379bba

New changeset 580cd9ab2992b7df6f4815020b5841e14a5a6977 by Brandt Bucher in branch 'main':
bpo-46072: Add detailed failure stats for BINARY_OP (GH-31289)
https://github.com/python/cpython/commit/580cd9ab2992b7df6f4815020b5841e14a5a6977