diff --git a/docs/html/md__cpp_usage.html b/docs/html/md__cpp_usage.html index 1163e5e15..640639cac 100644 --- a/docs/html/md__cpp_usage.html +++ b/docs/html/md__cpp_usage.html @@ -101,7 +101,15 @@ $(document).ready(function(){initNavTree('md__cpp_usage.html','');});
auto inv = monster->inventory();
assert(inv);
assert(inv->Get(9) == 9);
-

Direct memory access

+

Storing maps / dictionaries in a FlatBuffer

+

FlatBuffers doesn't support maps natively, but there is support to emulate their behavior with vectors and binary search, which means you can have fast lookups directly from a FlatBuffer without having to unpack your data into a std::map or similar.

+

To use it:

+

Direct memory access

As you can see from the above examples, all elements in a buffer are accessed through generated accessors. This is because everything is stored in little endian format on all platforms (the accessor performs a swap operation on big endian machines), and also because the layout of things is generally not known to the user.

For structs, layout is deterministic and guaranteed to be the same accross platforms (scalars are aligned to their own size, and structs themselves to their largest member), and you are allowed to access this memory directly by using sizeof() and memcpy on the pointer to a struct, or even an array of structs.

To compute offsets to sub-elements of a struct, make sure they are a structs themselves, as then you can use the pointers to figure out the offset without having to hardcode it. This is handy for use of arrays of structs with calls like glVertexAttribPointer in OpenGL or similar APIs.

diff --git a/docs/html/md__schemas.html b/docs/html/md__schemas.html index f7c39efb1..de755d06c 100644 --- a/docs/html/md__schemas.html +++ b/docs/html/md__schemas.html @@ -146,6 +146,7 @@ root_type Monster;
  • force_align: size (on a struct): force the alignment of this struct to be something higher than what it is naturally aligned to. Causes these structs to be aligned to that amount inside a buffer, IF that buffer is allocated with that alignment (which is not necessarily the case for buffers accessed directly inside a FlatBufferBuilder).
  • bit_flags (on an enum): the values of this field indicate bits, meaning that any value N specified in the schema will end up representing 1<<N, or if you don't specify values at all, you'll get the sequence 1, 2, 4, 8, ...
  • nested_flatbuffer: "table_name" (on a field): this indicates that the field (which must be a vector of ubyte) contains flatbuffer data, for which the root type is given by table_name. The generated code will then produce a convenient accessor for the nested FlatBuffer.
    • +
  • key (on a field): this field is meant to be used as a key when sorting a vector of the type of table it sits in. Can be used for in-place binary search.

    • JSON Parsing

    The same parser that parses the schema declarations above is also able to parse JSON objects that conform to this schema. So, unlike other JSON parsers, this parser is strongly typed, and parses directly into a FlatBuffer (see the compiler documentation on how to do this from the command line, or the C++ documentation on how to do this at runtime).

    diff --git a/docs/source/CppUsage.md b/docs/source/CppUsage.md index 91332d282..3bd706970 100755 --- a/docs/source/CppUsage.md +++ b/docs/source/CppUsage.md @@ -157,6 +157,32 @@ Similarly, we can access elements of the inventory array: assert(inv->Get(9) == 9); ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +### Storing maps / dictionaries in a FlatBuffer + +FlatBuffers doesn't support maps natively, but there is support to +emulate their behavior with vectors and binary search, which means you +can have fast lookups directly from a FlatBuffer without having to unpack +your data into a `std::map` or similar. + +To use it: +- Designate one of the fields in a table as they "key" field. You do this + by setting the `key` attribute on this field, e.g. + `name:string (key)`. + You may only have one key field, and it must be of string or scalar type. +- Write out tables of this type as usual, collect their offsets in an + array or vector. +- Instead of `CreateVector`, call `CreateVectorOfSortedTables`, + which will first sort all offsets such that the tables they refer to + are sorted by the key field, then serialize it. +- Now when you're accessing the FlatBuffer, you can use `Vector::LookupByKey` + instead of just `Vector::Get` to access elements of the vector, e.g.: + `myvector->LookupByKey("Fred")`, which returns a pointer to the + corresponding table type, or `nullptr` if not found. + `LookupByKey` performs a binary search, so should have a similar speed to + `std::map`, though may be faster because of better caching. `LookupByKey` + only works if the vector has been sorted, it will likely not find elements + if it hasn't been sorted. + ### Direct memory access As you can see from the above examples, all elements in a buffer are diff --git a/docs/source/Schemas.md b/docs/source/Schemas.md index 7f8b0a144..daeb90ac2 100755 --- a/docs/source/Schemas.md +++ b/docs/source/Schemas.md @@ -275,6 +275,9 @@ Current understood attributes: (which must be a vector of ubyte) contains flatbuffer data, for which the root type is given by `table_name`. The generated code will then produce a convenient accessor for the nested FlatBuffer. +- `key` (on a field): this field is meant to be used as a key when sorting + a vector of the type of table it sits in. Can be used for in-place + binary search. ## JSON Parsing diff --git a/include/flatbuffers/flatbuffers.h b/include/flatbuffers/flatbuffers.h index 88ea9ad0e..df7186327 100644 --- a/include/flatbuffers/flatbuffers.h +++ b/include/flatbuffers/flatbuffers.h @@ -288,6 +288,31 @@ public: return reinterpret_cast(&length_ + 1); } + template return_type LookupByKey(K key) const { + auto span = size(); + uoffset_t start = 0; + // Perform binary search for key. + while (span) { + // Compare against middle element of current span. + auto middle = span / 2; + auto table = Get(start + middle); + auto comp = table->KeyCompareWithValue(key); + if (comp > 0) { + // Greater than. Adjust span and try again. + span = middle; + } else if (comp < 0) { + // Less than. Adjust span and try again. + middle++; + start += middle; + span -= middle; + } else { + // Found element. + return table; + } + } + return nullptr; // Key not found. + } + protected: // This class is only used to access pre-existing data. Don't ever // try to construct these manually. @@ -304,6 +329,10 @@ template static inline size_t VectorLength(const Vector *v) { struct String : public Vector { const char *c_str() const { return reinterpret_cast(Data()); } + + bool operator <(const String &o) const { + return strcmp(c_str(), o.c_str()) < 0; + } }; // Simple indirection for buffer allocation, to allow this to be overridden @@ -646,6 +675,40 @@ class FlatBufferBuilder FLATBUFFERS_FINAL_CLASS { return Offset>(EndVector(len)); } + template Offset> CreateVector(const std::vector &v) { + return CreateVector(v.data(), v.size()); + } + + template Offset> CreateVectorOfStructs( + const T *v, size_t len) { + NotNested(); + StartVector(len * sizeof(T) / AlignOf(), AlignOf()); + PushBytes(reinterpret_cast(v), sizeof(T) * len); + return Offset>(EndVector(len)); + } + + template Offset> CreateVectorOfStructs( + const std::vector &v) { + return CreateVectorOfStructs(v.data(), v.size()); + } + + template Offset>> CreateVectorOfSortedTables( + Offset *v, size_t len) { + std::sort(v, v + len, + [this](const Offset &a, const Offset &b) -> bool { + auto table_a = reinterpret_cast(buf_.data_at(a.o)); + auto table_b = reinterpret_cast(buf_.data_at(b.o)); + return table_a->KeyCompareLessThan(table_b); + } + ); + return CreateVector(v, len); + } + + template Offset>> CreateVectorOfSortedTables( + std::vector *v) { + return CreateVectorOfSortedTables(v->data(), v->size()); + } + // Specialized version for non-copying use cases. Write the data any time // later to the returned buffer pointer `buf`. uoffset_t CreateUninitializedVector(size_t len, size_t elemsize, @@ -662,23 +725,6 @@ class FlatBufferBuilder FLATBUFFERS_FINAL_CLASS { reinterpret_cast(buf)); } - template Offset> CreateVector(const std::vector &v) { - return CreateVector(v.data(), v.size()); - } - - template Offset> CreateVectorOfStructs( - const T *v, size_t len) { - NotNested(); - StartVector(len * sizeof(T) / AlignOf(), AlignOf()); - PushBytes(reinterpret_cast(v), sizeof(T) * len); - return Offset>(EndVector(len)); - } - - template Offset> CreateVectorOfStructs( - const std::vector &v) { - return CreateVectorOfStructs(v.data(), v.size()); - } - static const size_t kFileIdentifierLength = 4; // Finish serializing a buffer by writing the root offset. diff --git a/include/flatbuffers/idl.h b/include/flatbuffers/idl.h index eccd1d84f..ffcef78b8 100644 --- a/include/flatbuffers/idl.h +++ b/include/flatbuffers/idl.h @@ -186,11 +186,14 @@ struct Definition { }; struct FieldDef : public Definition { - FieldDef() : deprecated(false), required(false), padding(0), used(false) {} + FieldDef() : deprecated(false), required(false), key(false), padding(0), + used(false) {} Value value; - bool deprecated; - bool required; + bool deprecated; // Field is allowed to be present in old data, but can't be + // written in new data nor accessed in new code. + bool required; // Field must always be present. + bool key; // Field functions as a key for creating sorted vectors. size_t padding; // Bytes to always pad after this field. bool used; // Used during JSON parsing to check for repeated fields. }; @@ -200,6 +203,7 @@ struct StructDef : public Definition { : fixed(false), predecl(true), sortbysize(true), + has_key(false), minalign(1), bytesize(0) {} @@ -214,6 +218,7 @@ struct StructDef : public Definition { bool fixed; // If it's struct, not a table. bool predecl; // If it's used before it was defined. bool sortbysize; // Whether fields come in the declaration or size order. + bool has_key; // It has a key field. size_t minalign; // What the whole object needs to be aligned to. size_t bytesize; // Size if fixed. }; @@ -271,6 +276,7 @@ class Parser { namespaces_.push_back(new Namespace()); known_attributes_.insert("deprecated"); known_attributes_.insert("required"); + known_attributes_.insert("key"); known_attributes_.insert("id"); known_attributes_.insert("force_align"); known_attributes_.insert("bit_flags"); diff --git a/src/idl_gen_cpp.cpp b/src/idl_gen_cpp.cpp index 0ab595c47..15b51513c 100644 --- a/src/idl_gen_cpp.cpp +++ b/src/idl_gen_cpp.cpp @@ -245,6 +245,24 @@ static void GenTable(const Parser &parser, StructDef &struct_def, code += "_nested_root() { return flatbuffers::GetRoot<"; code += nested_root->name + ">(" + field.name + "()->Data()); }\n"; } + // Generate a comparison function for this field if it is a key. + if (field.key) { + code += " bool KeyCompareLessThan(const " + struct_def.name; + code += " *o) const { return "; + if (field.value.type.base_type == BASE_TYPE_STRING) code += "*"; + code += field.name + "() < "; + if (field.value.type.base_type == BASE_TYPE_STRING) code += "*"; + code += "o->" + field.name + "(); }\n"; + code += " int KeyCompareWithValue("; + if (field.value.type.base_type == BASE_TYPE_STRING) { + code += "const char *val) const { return strcmp(" + field.name; + code += "()->c_str(), val); }\n"; + } else { + code += GenTypeBasic(parser, field.value.type, false); + code += " val) const { return " + field.name + "() < val ? -1 : "; + code += field.name + "() > val; }\n"; + } + } } } // Generate a verifier function that can check a buffer from an untrusted diff --git a/src/idl_parser.cpp b/src/idl_parser.cpp index 705e2a636..f7dd6acfd 100644 --- a/src/idl_parser.cpp +++ b/src/idl_parser.cpp @@ -395,6 +395,17 @@ void Parser::ParseField(StructDef &struct_def) { if (field.required && (struct_def.fixed || IsScalar(field.value.type.base_type))) Error("only non-scalar fields in tables may be 'required'"); + field.key = field.attributes.Lookup("key") != nullptr; + if (field.key) { + if (struct_def.has_key) + Error("only one field may be set as 'key'"); + struct_def.has_key = true; + if (!IsScalar(field.value.type.base_type)) { + field.required = true; + if (field.value.type.base_type != BASE_TYPE_STRING) + Error("'key' field must be string or scalar type"); + } + } auto nested = field.attributes.Lookup("nested_flatbuffer"); if (nested) { if (nested->type.base_type != BASE_TYPE_STRING) diff --git a/tests/monster_test.fbs b/tests/monster_test.fbs index 0ffd3dc15..a29b5e613 100755 --- a/tests/monster_test.fbs +++ b/tests/monster_test.fbs @@ -30,7 +30,7 @@ table Monster { pos:Vec3 (id: 0); hp:short = 100 (id: 2); mana:short = 150 (id: 1); - name:string (id: 3, required); + name:string (id: 3, required, key); color:Color = Blue (id: 6); inventory:[ubyte] (id: 5); friendly:bool = false (deprecated, priority: 1, id: 4); diff --git a/tests/monster_test_generated.h b/tests/monster_test_generated.h index c33ae57bd..2fedc9fa5 100755 --- a/tests/monster_test_generated.h +++ b/tests/monster_test_generated.h @@ -125,6 +125,8 @@ struct Monster FLATBUFFERS_FINAL_CLASS : private flatbuffers::Table { int16_t mana() const { return GetField(6, 150); } int16_t hp() const { return GetField(8, 100); } const flatbuffers::String *name() const { return GetPointer(10); } + bool KeyCompareLessThan(const Monster *o) const { return *name() < *o->name(); } + int KeyCompareWithValue(const char *val) const { return strcmp(name()->c_str(), val); } const flatbuffers::Vector *inventory() const { return GetPointer *>(14); } Color color() const { return static_cast(GetField(16, 8)); } Any test_type() const { return static_cast(GetField(18, 0)); } diff --git a/tests/test.cpp b/tests/test.cpp index 1abbfdcdd..5b55926c0 100644 --- a/tests/test.cpp +++ b/tests/test.cpp @@ -81,10 +81,19 @@ std::string CreateFlatBufferTest() { // create monster with very few fields set: // (same functionality as CreateMonster below, but sets fields manually) + flatbuffers::Offset mlocs[3]; auto fred = builder.CreateString("Fred"); - MonsterBuilder mb(builder); - mb.add_name(fred); - auto mloc2 = mb.Finish(); + auto barney = builder.CreateString("Barney"); + auto wilma = builder.CreateString("Wilma"); + MonsterBuilder mb1(builder); + mb1.add_name(fred); + mlocs[0] = mb1.Finish(); + MonsterBuilder mb2(builder); + mb2.add_name(barney); + mlocs[1] = mb2.Finish(); + MonsterBuilder mb3(builder); + mb3.add_name(wilma); + mlocs[2] = mb3.Finish(); // Create an array of strings: flatbuffers::Offset strings[2]; @@ -92,12 +101,12 @@ std::string CreateFlatBufferTest() { strings[1] = builder.CreateString("fred"); auto vecofstrings = builder.CreateVector(strings, 2); - // Create an array of tables: - auto vecoftables = builder.CreateVector(&mloc2, 1); + // Create an array of sorted tables, can be used with binary search when read: + auto vecoftables = builder.CreateVectorOfSortedTables(mlocs, 3); // shortcut for creating monster with all fields set: auto mloc = CreateMonster(builder, &vec, 150, 80, name, inventory, Color_Blue, - Any_Monster, mloc2.Union(), // Store a union. + Any_Monster, mlocs[1].Union(), // Store a union. testv, vecofstrings, vecoftables, 0); FinishMonsterBuffer(builder, mloc); @@ -163,9 +172,15 @@ void AccessFlatBufferTest(const std::string &flatbuf) { // Example of accessing a vector of tables: auto vecoftables = monster->testarrayoftables(); - TEST_EQ(vecoftables->Length(), 1U); + TEST_EQ(vecoftables->Length(), 3U); for (auto it = vecoftables->begin(); it != vecoftables->end(); ++it) - TEST_EQ(strcmp(it->name()->c_str(), "Fred"), 0); + TEST_EQ(strlen(it->name()->c_str()) >= 4, true); + TEST_EQ(strcmp(vecoftables->Get(0)->name()->c_str(), "Barney"), 0); + TEST_EQ(strcmp(vecoftables->Get(1)->name()->c_str(), "Fred"), 0); + TEST_EQ(strcmp(vecoftables->Get(2)->name()->c_str(), "Wilma"), 0); + TEST_NOTNULL(vecoftables->LookupByKey("Barney")); + TEST_NOTNULL(vecoftables->LookupByKey("Fred")); + TEST_NOTNULL(vecoftables->LookupByKey("Wilma")); // Since Flatbuffers uses explicit mechanisms to override the default // compiler alignment, double check that the compiler indeed obeys them: