Fork us on GitHub Follow us on Facebook Follow us on Twitter

Changes between Version 5 and Version 6 of StructuredBinaryData


Ignore:
Timestamp:
2012-05-08T04:57:59Z (7 years ago)
Author:
Sean Bartell
Comment:

Requirements and Existing Tools

Legend:

Unmodified
Added
Removed
Modified
  • StructuredBinaryData

    v5 v6  
    88[https://www.google-melange.com/gsoc/project/google/gsoc2012/wtachi/46005 GSoC project page].
    99
     10== Requirements ==
     11
     12* View on different levels; for instance, view the integer and sequence of
     13  bytes comprising a string if necessary.
     14* Check whether files are consistent.
     15* Handle broken files.
     16* Don’t try to read the whole file at once.
     17* Allow full modifications. Ideally, allow creation of a whole filesystem from scratch.
     18
    1019== Existing Tools ==
    1120
     
    1423=== [http://construct.wikispaces.com/ Construct] ===
    1524
    16 TODO: looks promising. Also look at issues and forks.
     25A Python library for creating declarative structure definitions. Each instance
     26of the `Construct` class has a name, and knows how to read from a stream, write
     27to a stream, and determine its length. Some predefined `Construct` subclasses
     28use an arbitrary Python function evaluated at runtime, or behave differently
     29depending on whether sub‐`Construct`s throw exceptions. `Const` uses a
     30sub‐`Construct` and makes sure the value is correct. Also has lazy
     31`Construct`s.
     32
     33Unfortunately, if you change the size of a structure, you still have to change
     34everything else manually.
     35
     36TODO: look at issues and forks.
    1737
    1838=== [http://bindata.rubyforge.org/ BinData] ===
    1939
    20 TODO: looks promising.
     40Makes good use of Ruby syntax; mostly has the same features as Construct.
    2141
    22 === [http://wsgd.free.fr/ Wireshark Generic Dissector] ===
     42=== Imperative DSLs ===
    2343
    24 The length and real value of a field can depend on all previous fields and use
    25 complex expressions. Structures can contain `if`/`while`/`continue`/`break`/…
    26 statements.
     44DSLs in this category are used in an obvious, deterministic manner, and complex
     45structures can’t be edited. They are simple imperative languages in which
     46fields, structures, bitstructures, and arrays can be defined. The length,
     47decoded value, and presence of fields can be determined by expressions using
     48any previously decoded field, and structures can use
     49`if`/`while`/`continue`/`break` and similar statements. Structures can inherit
     50from other structures, meaning that the parent’s fields are present at the
     51beginning of the child. Statements can move to a different offset in the input
     52data. There may be a real programming language that can be used along with the
     53DSL.
     54
     55 [http://pyffi.sourceforge.net/ PyFFI]::
     56  Lets you create or modify files instead of just reading them. Fields can
     57  refer to blocks of data elsewhere in the file. Uses an XML format.
     58 [http://www.synalysis.net/ Synalize It!]::
     59  Not completely imperative; if you declare optional structs where part of the
     60  data is constant, the correct struct will be displayed. Has a Graphviz export
     61  of file structure. Uses an XML format.
     62 Other free::
     63  [http://wsgd.free.fr/ Wireshark Generic Dissector].
     64 Other proprietary::
     65  [http://www.hhdsoftware.com/doc/hex-editor/language-reference-overview.html Hex Editor Neo].
    2766
    2867=== Less interesting tools ===
    2968
     69 Simple formats in hex editors::
     70  These support static fields and dynamic lengths only:
     71  [http://www.flexhex.com/ FlexHex],
     72  [http://hexedit.com/ HexEdit],
     73  [http://www.hexworkshop.com/ Hex Workshop], and
     74  [http://kde.org/applications/utilities/okteta/ Okteta].
     75 Simple formats elsewhere::
     76  [http://ff-extractor.sourceforge.net/ ffe],
     77  [http://bigeasy.github.com/node-packet/ Node Packet], and
     78  [https://www.secdev.org/projects/scapy/ Scapy]
     79  can only handle trivial structures.
     80  [http://docs.python.org/library/struct.html Python’s struct] and
     81  [https://github.com/ToxicFrog/vstruct VStruct]
     82  use concise string formats to describe simple structures.
     83  [https://bitbucket.org/haypo/hachoir Hachoir]
     84  uses Python for most things.
     85 Protocol definition formats::
     86  [https://en.wikipedia.org/wiki/Abstract_Syntax_Notation_One ASN.1],
     87  [https://en.wikipedia.org/wiki/Microsoft_Interface_Definition_Language MIDL],
     88  [http://piqi.org/ Piqi],
     89  and other IPC implementations go in the other direction: they generate a
     90  binary format from a text description of a structure. ASN.1 in particular
     91  has many features.
    3092 [https://www.wireshark.org/ Wireshark] and [http://www.tcpdump.org/ tcpdump]::
    3193  As the Construct wiki notes, you would expect these developers to have some
    3294  sort of DSL, but they just use C for everything. Wireshark does use ASN.1,
    3395  Diameter, and MIDL for protocols developed with them.
    34  [http://kde.org/applications/utilities/okteta/ Okteta]::
    35   Has an XML format for simple structures, where the length of a field can
    36   depend on a previous value. Also has an on‐line database of structures, but
    37   it isn’t very popular—there are only nine submissions!
    38  Other simple formats::
    39   [http://ff-extractor.sourceforge.net/ ffe] can only handle trivial
    40   structures. [http://docs.python.org/library/struct.html Python’s struct]
    41   and [https://github.com/ToxicFrog/vstruct VStruct] use concise string formats
    42   to describe simple structures.
    43  Other hex editors::
    44   [http://beye.sourceforge.net/ Beye], [http://home.gna.org/bless/ Bless], and
    45   [https://live.gnome.org/Ghex GHex] lack interesting features.
    46  Protocol definition formats::
    47   [https://en.wikipedia.org/wiki/Abstract_Syntax_Notation_One ASN.1],
    48   [https://en.wikipedia.org/wiki/Microsoft_Interface_Definition_Language MIDL],
    49   and other IPC implementations go in the other direction: they generate a
    50   binary format from a text description of a structure. ASN.1 in particular
    51   has many features.