python-bootcamp

Table of Contents

Output:

Answer: Yes, the attribute name of the loader object can be used to determine if the loader is source, builtin, or extension.
Answer:

setup.py:

1. Run the module-inspector script, and fix any problems that prevent it from running.

2. Look at the output. Notice that some of the module names seem similar, and are cached via two different loader methods.

3. Based on what you see from the module-inspector script, modify your answer from question 10 from the day 9 homework, so the similar module names aren't counted multiple times.

import sys
import os
from collections import defaultdict

def get_module_stats():
    byte_count = 0
    dso_count = 0
    py_count = 0
    module_files = set()
    module_dirs = defaultdict(int)

    for name, module in sys.modules.items():
        if module is None:
            continue

        filename = getattr(module, '__file__', None)
        if not filename:
            continue

        if filename.endswith('.pyc') or filename.endswith('.pyo'):
            byte_count += 1
        elif os.path.splitext(filename)[-1] in ('.so', '.dll'):
            dso_count += 1
        else:
            py_count += 1

        # check if module filename has been counted already
        if filename not in module_files:
            module_files.add(filename)

            module_dir = os.path.dirname(os.path.abspath(filename))
            module_dirs[module_dir] += 1

    total_count = byte_count + dso_count + py_count

    byte_percent = byte_count / total_count * 100
    dso_percent = dso_count / total_count * 100
    py_percent = py_count / total_count * 100

    return {
        'bytecode': {
            'count': byte_count,
            'percent': byte_percent,
        },
        'DSO': {
            'count': dso_count,
            'percent': dso_percent,
        },
        'python': {
            'count': py_count,
            'percent': py_percent,
        },
        'total': total_count,
        'module_dirs': module_dirs,
    }

def print_module_stats():
    stats = get_module_stats()

    print(f'Total modules: {stats["total"]}')
    print(f'Bytecode modules: {stats["bytecode"]["count"]} ({stats["bytecode"]["percent"]:.2f}%)')
    print(f'DSO modules: {stats["DSO"]["count"]} ({stats["DSO"]["percent"]:.2f}%)')
    print(f'Python modules: {stats["python"]["count"]} ({stats["python"]["percent"]:.2f}%)')

    print('\nModule directory histogram:')
    for module_dir, count in sorted(stats['module_dirs'].items(), key=lambda x: x[1], reverse=True):
        print(f'{module_dir}: {"*" * count}')

if __name__ == '__main__':
    print_module_stats()

By using a set called module_files to keep track of the filenames of the modules that we have counted so far. Before counting a module, we check if its filename has been counted already. If it has, we skip the module and move on to the next one. This ensures that similar module names are counted only once.

Output:

Total modules: 1011
Bytecode modules: 0 (0.00%)
DSO modules: 51 (5.04%)
Python modules: 960 (94.96%)

Module directory histogram:
/Users/sloetjeswong/opt/anaconda3/lib/python3.9: ***************************************************************************************************************
/Users/sloetjeswong/opt/anaconda3/lib/python3.9/site-packages/debugpy/_vendored/pydevd/_pydevd_bundle: ************************************************
/Users/sloetjeswong/opt/anaconda3/lib/python3.9/site-packages/IPython/core: *******************************************
/Users/sloetjeswong/opt/anaconda3/lib/python3.9/lib-dynload: **************************************
/Users/sloetjeswong/opt/anaconda3/lib/python3.9/site-packages/IPython/utils: ********************************
/Users/sloetjeswong/opt/anaconda3/lib/python3.9/asyncio: **************************
/Users/sloetjeswong/opt/anaconda3/lib/python3.9/site-packages/jedi/inference: **********************
/Users/sloetjeswong/opt/anaconda3/lib/python3.9/site-packages/setuptools/_distutils: *******************
/Users/sloetjeswong/opt/anaconda3/lib/python3.9/site-packages/setuptools: ******************
/Users/sloetjeswong/opt/anaconda3/lib/python3.9/site-packages/jupyter_client: *****************
/Users/sloetjeswong/opt/anaconda3/lib/python3.9/email: *****************
/Users/sloetjeswong/opt/anaconda3/lib/python3.9/site-packages/ipykernel: ****************
/Users/sloetjeswong/opt/anaconda3/lib/python3.9/site-packages/prompt_toolkit: ****************
/Users/sloetjeswong/opt/anaconda3/lib/python3.9/site-packages/IPython/core/magics: ***************
...

4. Inspect the loader attribute of each entry in the sys.modules dict. Can you find an attribute of loader which reliably tells you if the loader is source, builtin, or extension?

Answer: Yes, the attribute name of the loader object can be used to determine if the loader is source, builtin, or extension.

If the name attribute is 'built-in', it means that the module is a built-in module.
If the name attribute is 'extension', it means that the module is a compiled extension module.
Otherwise, the module is a source module.

5. Add import pandas to the top of the module-inspector script and re-run it. Notice that many of the pandas modules are not loaded using the SourceFileLoader. Can you explain why these modules are different?

Answer:

This is because Pandas is a large library with many sub-modules and sub-packages, and some of these sub-modules and sub-packages are implemented using compiled extensions written in C or C++. These extensions are loaded dynamically at runtime by Python's import machinery, and therefore they are not loaded using the SourceFileLoader like pure Python modules are.
When a Python module contains compiled extensions, Python uses a different type of loader called an "extension loader" to load the module. Extension loaders are responsible for loading compiled binary code that is specific to the operating system and architecture of the system running Python.
In the case of Pandas, many of the sub-modules and sub-packages are implemented using compiled extensions, and therefore they are loaded using an extension loader rather than the SourceFileLoader. This is why we see a mix of different types of loaders in the loader column when we run the module-inspector script on a system that has Pandas installed.

6. Extra credit: use the freeze utility to turn the module-inspector script into a real app.

setup.py:

import sys
from cx_Freeze import setup, Executable

build_exe_options = {"packages": ["os"], "excludes": ["tkinter"]}

setup(
    name="Module Inspector",
    version="1.0",
    description="A utility to inspect Python modules",
    options={"build_exe": build_exe_options},
    executables=[Executable("module-inspector.py", base=base)]
)