1. Use importlib.util.find_spec to create a function that takes a Python module or package name, and returns the total number of bytes in the cached .pyc file, or the total number of bytes in the .py file if there is no cached bytecode. You can use the os.stat command to get the size of a file.
import importlib.util
import os
def get_module_path(module_name):
module = sys.modules.get(module_name)
if module is None:
return None
if hasattr(sys, 'frozen') and sys.frozen:
# Module is part of a frozen binary or executable
return sys.executable
elif hasattr(module, '__file__'):
# Module has a __file__ attribute
return module.__file__
else:
# Module doesn't have a __file__ attribute
return None
def get_module_size(module_name):
module_path = get_module_path(module_name)
if module_path is None:
return 0
if module_path.endswith('.pyc'):
filename = module_path[:-1]
else:
filename = module_path
return os.stat(filename).st_size
2. Create another function that uses the last function to see how many bytes are in all the modules listed in sys.modules.
def get_all_modules_size():
total_size = 0
for module_name, module in sys.modules.items():
if not hasattr(module, '__file__'):
continue
module_path = get_module_path(module_name)
if module_path is not None:
module_size = get_module_size(module_name)
total_size += module_size
return total_size
print(get_all_modules_size())
3. Run your Python interpreter like you would normally, using Jupyter, and run the last function. How many bytes are being used?
4. Now, run the same function, but start the Python interpreter in Powershell (Windows), or the regular shell (MacOS/Linux) with the -E flag, which ignores environment variables and should only load the modules that came with the original build of the Python interpreter. How many bytes are being used now?
5. With only the sys module loaded, look at the sys.modules dict. What is the type of the value of the os key?
Answer:
-
The type of the value of the
oskey in the sys.modules dictionary depends on whether the os module has been loaded yet. -
If the
osmodule has not been loaded yet, the value of theoskey will beNoneType. -
If the os module has been loaded, the value of the os key will be a module object representing the
osmodule.
6. Import the os module. What happens when you type type(os)?
Answer:
- The
type()function returns the type of theosmodule, which is a module object.
7. Restart the Python interpreter. Can you figure out how to call os.getlogin() without saying import os?
Example:
login = __import__('os').getlogin()
print(login)
8. If you can, explain why this works and quote the relevant sections of section 5 in the language reference.
Answer:
- According to the Python language reference documentation on Importing Modules (section 5.6), the
__import__()function is a low-level interface for dynamically importing modules at runtime. The documentation states that:
When the name variable is of the form package.module, normally, the top-level package (the name up till the first dot) is returned, not the module named by name. However, when a non-empty fromlist argument is given, the module named by name is returned.
-
In the code above, the name of the module is
"os", and the fromlist argument is not given, so the top-level package"os"is returned, which is equivalent to importing os with a regular import statement. -
Therefore, the expression
__import__('os').getlogin()is equivalent toos.getlogin(), but it dynamically imports theosmodule at runtime instead of using a regular import statement.
9. Using the functions you created above and sys.modules, create a module designed to be run from the command line (python -m myscript.py) that generates a report summarizing the module environment, and break down the percentages of bytecode modules, DSO modules (these have file end in .so or .dll), and normal python code.
import sys
import os
def get_module_stats():
byte_count = 0
dso_count = 0
py_count = 0
for name, module in sys.modules.items():
if module is None:
continue
filename = getattr(module, '__file__', None)
if not filename:
continue
if filename.endswith('.pyc') or filename.endswith('.pyo'):
byte_count += 1
elif os.path.splitext(filename)[-1] in ('.so', '.dll'):
dso_count += 1
else:
py_count += 1
total_count = byte_count + dso_count + py_count
byte_percent = byte_count / total_count * 100
dso_percent = dso_count / total_count * 100
py_percent = py_count / total_count * 100
return {
'bytecode': {
'count': byte_count,
'percent': byte_percent,
},
'DSO': {
'count': dso_count,
'percent': dso_percent,
},
'python': {
'count': py_count,
'percent': py_percent,
},
'total': total_count,
}
def print_module_stats():
stats = get_module_stats()
print(f'Total modules: {stats["total"]}')
print(f'Bytecode modules: {stats["bytecode"]["count"]} ({stats["bytecode"]["percent"]:.2f}%)')
print(f'DSO modules: {stats["DSO"]["count"]} ({stats["DSO"]["percent"]:.2f}%)')
print(f'Python modules: {stats["python"]["count"]} ({stats["python"]["percent"]:.2f}%)')
if __name__ == '__main__':
print_module_stats()
Output:
Total modules: 1011
Bytecode modules: 0 (0.00%)
DSO modules: 51 (5.04%)
Python modules: 960 (94.96%)
10. Add a feature to the script which looks at file for each module, trims the filename to get the directory name, then prints out a histogram of how many modules are in each directory name. If you can't find a text histogram output library in anaconda's built-in modules, just print out the * character for each module.
import sys
import os
from collections import defaultdict
def get_module_stats():
byte_count = 0
dso_count = 0
py_count = 0
module_dirs = defaultdict(int)
for name, module in sys.modules.items():
if module is None:
continue
filename = getattr(module, '__file__', None)
if not filename:
continue
if filename.endswith('.pyc') or filename.endswith('.pyo'):
byte_count += 1
elif os.path.splitext(filename)[-1] in ('.so', '.dll'):
dso_count += 1
else:
py_count += 1
module_dir = os.path.dirname(os.path.abspath(filename))
module_dirs[module_dir] += 1
total_count = byte_count + dso_count + py_count
byte_percent = byte_count / total_count * 100
dso_percent = dso_count / total_count * 100
py_percent = py_count / total_count * 100
return {
'bytecode': {
'count': byte_count,
'percent': byte_percent,
},
'DSO': {
'count': dso_count,
'percent': dso_percent,
},
'python': {
'count': py_count,
'percent': py_percent,
},
'total': total_count,
'module_dirs': module_dirs,
}
def print_module_stats():
stats = get_module_stats()
print(f'Total modules: {stats["total"]}')
print(f'Bytecode modules: {stats["bytecode"]["count"]} ({stats["bytecode"]["percent"]:.2f}%)')
print(f'DSO modules: {stats["DSO"]["count"]} ({stats["DSO"]["percent"]:.2f}%)')
print(f'Python modules: {stats["python"]["count"]} ({stats["python"]["percent"]:.2f}%)')
print('\nModule directory histogram:')
for module_dir, count in sorted(stats['module_dirs'].items(), key=lambda x: x[1], reverse=True):
print(f'{module_dir}: {"*" * count}')
if __name__ == '__main__':
print_module_stats()