Python: Making ctypes Structures Beautiful

Python: Making ctypes Structures Beautiful(y4kstudios.com)

44 points by they4kman 13 years ago | 8 comments

densh 13 years ago |

People never seem to fully understand namespaces. You don't need any additional prefixes for classes and functions inside of your module. Its name already does separate it from everything else. Moreover "from module import *" is evil. Period.

import newstruct as ns

...

x = ns.long()

DeepDuh 13 years ago | |

Makes me think.. is the NS prefix in cocoa frameworks short for "newstruct"? I always wondered about that one.

spleeyah 13 years ago | | |

    > Cocoa classes begin with the acronym "NS" (standing either for the
    > NeXT-Sun creation of OpenStep, or for the original proprietary term
    > for the OpenStep framework, NeXTSTEP)

http://en.wikipedia.org/wiki/Cocoa_(API)#Cocoa_history

jessedhillon 13 years ago |

That's my module. Now, how can I make it better?

You could have those fields know their own sizes, and then you could have Struct subclasses know the size of their instances, so you don't have to call sizeof. E.g.:

    class DataHeader(Struct):
        address = newstruct.long()
        typ =     newstruct.byte()
        tag =     newstruct.short()
        data =    newstruct.byte(length=40) # 40 bytes

    >>> fp = open('mydata', 'rb')
    >>> dh = DataHeader.read(fp)
    # or perhaps: DataHeader.load(fp.read(len(dh)))
    # this way the side-effect of moving fp's current position is more explicit

    >>> fp.close()
    >>> len(dh)
    12345    # or whatever, sum of sizes for long, 41 bytes, and short

JoachimSchipper 13 years ago | |

Additional issues: is a long guaranteed to be 64 bits, or whatever the platform calls long? Is the structure padded (in the same way that the C compiler pads structs)? And, of course, which endianness is used?

jevinskie 13 years ago |

My go-to library for binary mucking in Python is the fantastic Construct library. [0] [1] It is a declarative way of specifying binary structures. It handles endianness, serialization, and deserialization for you. You can build structures of structures with Construct. I used it to parse the PS3's pseudo-ELF files (SELFs) and was quite successful except for a few shortcomings in the library. I don't blame Construct though because the SELF format is a wild beast. Corbin Simpson (the author) was quite helpful when I was looking into the possibility of fixing my issues. Look at the ELF parser to see how easy it is to use! [2]

[0]: http://construct.readthedocs.org/en/latest/

[1]: https://github.com/construct/construct

[2]: https://github.com/construct/construct/blob/master/construct...

wisesage5001 13 years ago |

One thing I really like about the struct module's format string is the ability to easily specify the endian-ness of the data. How does newstruct (or the underlying ctypes) handle endian-ness (if at all)?

veyron 13 years ago | |

endianness only matters when you bother interpreting the data. The storage concept shouldn't care -- after all, 8 bytes is 8 bytes is 8 bytes.

Struct needs it because it is interpreting data. unpack needs to know the endianness to correctly interpret numeric types, and pack needs to know the endianness to know how to serialize.

FYI: the C way of doing this is byteswapping.

class DataHeader(Struct): address = newstruct.long() typ = newstruct.byte() tag = newstruct.short() data = newstruct.byte(length=40) # 40 bytes >>> fp = open('mydata', 'rb') >>> dh = DataHeader.read(fp) # or perhaps: DataHeader.load(fp.read(len(dh))) # this way the side-effect of moving fp's current position is more explicit >>> fp.close() >>> len(dh) 12345 # or whatever, sum of sizes for long, 41 bytes, and short