Python ByteCoat Frequently Asked Questions
1. Are signals supported by ByteCoat?
In Python the interpreter loop periodically processes all pending events, such as UNIX signal handlers or Mac I/O completion routines. Currently, ByteCoat does not trigger the processing of those events. However, functions translated with ByteCoat will usually call other non-ByteCoat functions, such as built-in functions, that do process events. In general, events are processed with the next call to the Python interpreter loop.
In case of long running ByteCoat functions that do not respond to signals, insert a statement like
exec "pass"
into the long loop, that enables the processing of events.
2. What is the data type of a ByteCoat function?
Python functions translated with ByteCoat are not of the same data type as normal Python functions. Therefore, checks such as isinstance(foo,...), type(foo) and functions from the inspect module will not work as expected. An example is the functionality of the sip framework, that checks methods of derived classes with PyFunction_Check. This check fails and sip reports the error "Abstract functions not overwritten".
One possible solution is to decorate the corresponding functions with a special wrapper:
def wrapper(func):
exec \
"""
def helper(*args,**kwargs):
return func(*args, **kwargs)
""" in locals()
return helper
@wrapper
def foo(a,b):
return a+b
assert isinstance(foo,types.FunctionType)
3. I cannot access the byte code of a Function?
The objective of Python ByteCoat is to get rid of the byte code, so the original source code can not be reverse engineered. In case a user tries to access the func_code attribute of a ByteCoat function, an exception RuntimeError("code is not accessible") is raised.
4. Is tracing of ByteCoat functions supported?
In general tracing is not supported. If a tracing function is set with sys.settrace(foo), it will n ot be called for any tracing events inside functions translated by ByteCoat.
Python ByteCoat for Python 2.6 has an experimental feature for tracing inside ByteCoat functions. It can be enabled with the
-r or --tracing
command line option, but will result in much larger C files.
5. Can I delete and reload modules translated with ByteCoat?
Calling
import modname
del sys.modules["modname"]
import modname
will invalidate the globals() dictionary of the functions in module "modname". This is caused by the specific import mechanism of Python for extension modules. Extension modules are not reloaded but constructed from a hidden copy of the old globals dictionary. However, as soon as the reference count of the old module runs to 0, it sets all entries in the old globals dict to None. The functions o f the module still work with a reference to this old globals dict with invalid content.
6. Is the __new__ operator for ByteCoat functions implemented?
No, this operator is not implemented, so Python falls back to
object.__new__
which returns the following error message:
TypeError: object.__new__(function) is not safe, use function.__new__()
7. Warnings show confusing characters?
The internal _warnings module is responsible for displaying warning messages. It extracts the filename from the globals dictionary and the line number from the frame raising the warning and tries to display the corresponding source code. In case the line number option is enabled (-n) it gets the correct line number but the filename points to the shared object library. Therefore, the warnings shows a certain line in the .so file, resulting in arbitrary machine code to be displayed.
Normal exception tracebacks are not affected, since it extracts the filename from the frame which has been set to the original .py file. ByteCoat also sets the filename in the globals() dictionary to the .py file, but the import mechanism of Python will overwrite it with the .so filename.
In case the line numbering option is not specified, the filename is always the shared object file and the line number is -1. Hence, no source code is displayed in both warnings and tracebacks.
8. I get an internal compiler error with gcc?
The C files generated by ByteCoat can be very large. In some cases the C compiler fails to compile them and exits with an internal compiler error such as:
internal compiler error: in final_scan_insn, at final.c:1794
We have currently no solution for this, except splitting your python module into multiple files and translate them individually.
9. I get the error message "conflicting types for 'init'"?
An extension module always contains a function
init<modulename>
that represents the central entry point of the extension module. In case the assembled name collides with an identifier from any required C header, the above error message is printed. Known cases are listed below
state.py leads to declaration of initstate (declared inside stdlib.h)
groups.py leads to declaration of initgroups (declared inside unistd.h)
The only solution to this problem is avoiding the corresponding module names.
10. Packages with a translated __init__ module can not be imported?
If a __init__ module of package
ImportError: dynamic module does not define init function (init<pkg>)
This happens because the central entry function of __init__ modules must be called init
One possible workaround is renaming the function in the C file after generating it before compilation.
11. The __file__ attribute of a module may contain no directory information?
As soon as a regular Python module is imported, the importer (function PyImport_ExecCodeModuleEx o f the interpreter) first the globals dictionary. Afterwards, it sets the __file__ attribute to the correct and qualified path of the corresponding .py file. Following, the global scope of the module is executed and thus contains the right path in the __file__ attribute.
When importing a extension module such as the ones generated by ByteCoat, this order is reversed. First, the init function of the module is called. In case of a ByteCoat module the init function sets the __file__ attribute to the static name modulename.so (or modulename.py if command line option -n is given) and afterwards triggers the execution of the global scope. After completion o f the init function the importer (function _PyImport_LoadDynamicModule of the interpreter) sets the __file__ attribute to the correct path and therefore overwrites ByteCoat's static name. As a consequence, only the short file name is available in the global scope, but later on, e.g. during function calls and direct access to the globals dictionary, the correct long file name is set in the __file__ attribute.
FAQ 7 describes another undesirable consequence of the same problem.
The module file name inside the instances of frames of ByteCoat functions is not affected by this problem. Frames of ByteCoat functions always use the short static module filename. This short name then occurs in tracebacks that include ByteCoat functions.
A possible solution to avoid this problem is retrieving the __file__ attribute after the import of the extension module is completed:
myapp/__init.py
import mysub.mysubmodule
print mysub.mysubmodule.__file__ # correct path
mysub.mysubmodule.printFilename() # correct path
myapp/mysub/mysubmodule.py
print __file__ # short path
def printFilename():
print __file__
12. Some exceptions print different messages?
Sometimes, the interpreter prints different messages for an exception than the compiled ByteCoat Module. This is because ByteCoat uses a different type to represent functions.
FAQ 2 describes another unwanted behaviour because of this problem.
def a(x, **kws):
pass
Output of original python module:
Python:
>>> a(None, **{"x": None})
Traceback (most recent call last):
File "", line 1, in
TypeError: a() got multiple values for keyword argument 'x'
>>>
Output of compiled python module (ByteCoat):
Python:
>>> a(None, **{"x": None})
Traceback (most recent call last):
File "", line 1, in
TypeError: function object got multiple values for keyword argument 'x'
>>>