SVIG/python-bootcamp
7
0
Fork 1
Code Issues Pull Requests 1 Releases Wiki Activity

Update page 'Day 10 homework - Sloetjes '

xuanyi.wang
2023-04-19 19:35:58 +02:00
parent f29605ec1b
commit 8596ea5807
1 changed files with 4 additions and 93 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Day 10 homework - Sloetjes .md
+4 -93
@@ -114,99 +114,10 @@ Module directory histogram:
* Otherwise, the module is a source module.
**5. Add `import pandas` to the top of the module-inspector script and re-run it. Notice that many of the pandas modules are not loaded using the `SourceFileLoader`. Can you explain why these modules are different?**
```
import sys
import os
import importlib
from collections import defaultdict
import pandas as pd
def get_module_stats():
byte_count = 0
dso_count = 0
py_count = 0
module_dirs = defaultdict(int)
for name, module in sys.modules.items():
if module is None:
continue
filename = getattr(module, '__file__', None)
if not filename:
continue
if filename.endswith('.pyc') or filename.endswith('.pyo'):
byte_count += 1
elif os.path.splitext(filename)[-1] in ('.so', '.dll'):
dso_count += 1
else:
py_count += 1
module_dir = os.path.dirname(os.path.abspath(filename))
module_dirs[module_dir] += 1
total_count = byte_count + dso_count + py_count
byte_percent = byte_count / total_count * 100
dso_percent = dso_count / total_count * 100
py_percent = py_count / total_count * 100
return {
'bytecode': {
'count': byte_count,
'percent': byte_percent,
},
'DSO': {
'count': dso_count,
'percent': dso_percent,
},
'python': {
'count': py_count,
'percent': py_percent,
},
'total': total_count,
'module_dirs': module_dirs,
}
def print_module_stats():
stats = get_module_stats()
print(f'Total modules: {stats["total"]}')
print(f'Bytecode modules: {stats["bytecode"]["count"]} ({stats["bytecode"]["percent"]:.2f}%)')
print(f'DSO modules: {stats["DSO"]["count"]} ({stats["DSO"]["percent"]:.2f}%)')
print(f'Python modules: {stats["python"]["count"]} ({stats["python"]["percent"]:.2f}%)')
print('\nModule directory histogram:')
for module_dir, count in sorted(stats['module_dirs'].items(), key=lambda x: x[1], reverse=True):
print(f'{module_dir}: {"*" * count}')
if __name__ == '__main__':
pd_version = pd.__version__
print(f'Pandas version: {pd_version}')
print_module_stats()
```
###### Output:
```
Pandas version: 1.4.4
Total modules: 1422
Bytecode modules: 0 (0.00%)
DSO modules: 113 (7.95%)
Python modules: 1309 (92.05%)
Module directory histogram:
/Users/sloetjeswong/opt/anaconda3/lib/python3.9: *****************************************************************************************************************
/Users/sloetjeswong/opt/anaconda3/lib/python3.9/site-packages/debugpy/_vendored/pydevd/_pydevd_bundle: ************************************************
/Users/sloetjeswong/opt/anaconda3/lib/python3.9/site-packages/IPython/core: *******************************************
/Users/sloetjeswong/opt/anaconda3/lib/python3.9/lib-dynload: ****************************************
/Users/sloetjeswong/opt/anaconda3/lib/python3.9/site-packages/IPython/utils: ********************************
/Users/sloetjeswong/opt/anaconda3/lib/python3.9/site-packages/numpy/core: *****************************
/Users/sloetjeswong/opt/anaconda3/lib/python3.9/asyncio: **************************
/Users/sloetjeswong/opt/anaconda3/lib/python3.9/site-packages/pandas/_libs: ************************
/Users/sloetjeswong/opt/anaconda3/lib/python3.9/site-packages/jedi/inference: **********************
/Users/sloetjeswong/opt/anaconda3/lib/python3.9/site-packages/numpy/lib: **********************
/Users/sloetjeswong/opt/anaconda3/lib/python3.9/site-packages/pandas/core: **********************
/Users/sloetjeswong/opt/anaconda3/lib/python3.9/site-packages/setuptools/_distutils: ********************
...
```
##### Answer:
* This is because Pandas is a large library with many sub-modules and sub-packages, and some of these sub-modules and sub-packages are implemented using compiled extensions written in C or C++. These extensions are loaded dynamically at runtime by Python's import machinery, and therefore they are not loaded using the SourceFileLoader like pure Python modules are.
* When a Python module contains compiled extensions, Python uses a different type of loader called an "extension loader" to load the module. Extension loaders are responsible for loading compiled binary code that is specific to the operating system and architecture of the system running Python. These loaders are typically implemented using the operating system's dynamic loading facilities (e.g., dlopen() on Unix-like systems, or LoadLibrary() on Windows).
* In the case of Pandas, many of the sub-modules and sub-packages are implemented using compiled extensions, and therefore they are loaded using an extension loader rather than the SourceFileLoader. This is why you see a mix of different types of loaders in the loader column when you run the module-inspector script on a system that has Pandas installed.
**Extra credit: use the freeze utility to turn the module-inspector script into a real app.**