Limit the length of the fuzzer input to 16384 characters to prevent timeout
in JSON parser (Vector of tables with key).
Related oss-fuzz issues:
- 5742497110294528
* Remove a lot of redundancy from the Python generated code.
Update tutorial to reflect new Python generated code.
* Add aliases for newly deprecated Python generated methods.
This should help with backwards compatibility.
* Fix incorrect names in deprecated comments.
* remove inner attributes
* Added test for outdir in Rust
* add bin/outdir
* Moved outdir test to its own package and only run it if flatc is available
Co-authored-by: Casper Neo <cneo@google.com>
* Add codegen test for namespaced unions
* [Rust] Handle cross-namespace union use
* [Rust] Test namespace handling
* [Rust] Drop trailing whitespace in codegen
* [Rust] Set flags in generate_code.bat to match .sh
* [C#] Add additional namespace test file
* Define presence.
* Migrate to IsRequired and IsOptional methods
* moved stuff around
* Removed optional and required bools from FieldDef
* change assert to return error
* Fix tests.cpp
* MakeFieldPresence helper
* fmt
* old c++ compatibility stuff
Co-authored-by: Casper Neo <cneo@google.com>
* TS/ES6 modules spike iteration 1
* Initial modularized dasherized output
* Remove obsoleted parts and namespace wrapping
* Use _flatbuffers_ prefix
* First part of imports logic
* Second part of imports logic
* Fix TS/JS code removal mixup
* Alias imported symbols if same name from different namespaces and some fixes
* Use star import for bare imports
* Fix messed up string concat
* var to const and remove not needed semi
* Remove some cases of ns prefixing
* Add missing space
* Cleanups
* Completed initial import tracking logic
* Compilable output
* Adjust TypeScriptTest and dependents to work
* Use local flatbuffers package for tests
* Refactor away use of any
* Remove obsolete imported_fileset and reexport_map
* Still need any and fix JavaScriptTest.sh
* Fix test runs out of the box
* Temp add generated files
* TypeScriptTest replaces JavaScriptTest and cleanups
* Also remove reference to JavaScriptTest in TestAll.sh
* Remove old generated ts/js files
* Remove use of --js in generate_code scripts
* idl_gen_js_ts to idl_gen_ts and removal of js gen
* Remove obsoleted options
* Fix obsolete ts test detection
* Tweak ts compilation be as strict as possible
* Remove jsdoc type annotation generation
* Generated test ts files
* Fix search and replace messup
* Regenerated ts test output
* Use CharToLower
* Use normal for loop
* Rework namespacedir
* Revert "Rework namespacedir"
This reverts commit 6f4eb0104ceeb86011bb076ebca901138c48e068.
* Revert "Use normal for loop"
This reverts commit 676b2135bfaa1853dfbb06c92b5c16a0d81bb13a.
* Revert "Use CharToLower"
This reverts commit 2d08648d0d72d0af201fad80d54cdc76412b35e9.
* Again do rework but correct
* Avoid runtime cast
* Fix test runs
* Also add npm install to get tsc
* Bump node test versions
* for range to std for loop
* Clang format
* Missed one clang format
* Move accessor to later
* Attempt to make windows version of TypeScriptTest
* Want to see the output
* Try to get newer node at appveyor
* Style changes
* Unset FieldDef.optional if its key
* removed StringKey table, just removed an extra "required, key"
* removed extra newline
Co-authored-by: Casper Neo <cneo@google.com>
* Fix miri problems by assuming alignment is 1 in rust
* Removed is_aligned fn from rust verifier.
* Add back is_aligned, but make it w.r.t. buffer[0]
* touch unused variable
* touch unused variable
* +nightly
* Move Rust miri testing into its own docker
* fix bash
* missing one endian conversion
* fix endianness2
* format stuff
Co-authored-by: Casper Neo <cneo@google.com>
For some reason, this fuzzer failed to load the binary schema file
when run on the `/clusterfuzz` server.
Issue: https://oss-fuzz.com/testcase-detail/6215075358703616
This issue doesn't reproduce locally with the latest oss-fuzz docker image.
* tests/GoTest.sh: Fix flags.Parse location to work on new go SDKs.
Calling flags.Parse() within init() races with other packages which register
flags in their init(), and in particular with the testing package itself. It is
more reliable to call flags.Parse() from a TestMain implementation.
See https://github.com/golang/go/issues/31859,
https://github.com/golang/go/issues/33869.
* .github: Enable build-go action in build.yaml workflow.
This commit fixes handling of default and NULL `key` fields in `Parser::ParseVector` (#5928).
The JSON generator updated. It outputs `key` fields even if the `--force-defaults` option is inactive.
Additional test cases for `key` added.
* [idl_parser] Improve stack overflow protection
Add stack overflow protection for Flexbuffer and nested Flatbuffer parsers.
Replaces the `Recurse()` method by the new ParseDepthGuard RAII class.
* Remove move operator from Parser.
It was wrong decision to add move ctor and assignment into Parser class.
These operators will make it extremely difficult to add constant or reference fields in the future.
* Remove ';' from definition of FLATBUFFERS_DELETE_FUNC
* Format code
* Make this PR compatible with MSVC2010 (it doesn't support inherited ctor)
* [idl_parser] Check the range of explicitly set field's id value
The explicitly set `id` attribute should be a non-negative value of the `voffset_t` type.
* Format FieldIdentifierTest()
* Adds shared strings and tests for shared strings
* Adds resets on string_map
* Moved shared strings to use vector instead of hashmap
* Addresses all the issues
* Resolves some comments
* Rebuild the way swift handles structs from scratch
* Updates docs, and sample binary
* Replaces InMemory to Mutable
* Migrates docs from inmemory
* use inline for some functions
* Renamed Mutable objects
* Updates documentation
The infinityf symbol is causing a conflict when building for cygwin. In
the cygwin math.h header there is also a symbol called infinityf. So
this patch is needed to be able to build the flatbuffer tests in a
cygwin environment.
* Add vectorNumElements attribute to Builder for simpler vector creation.
This adds a default to EndVector which should simplify its use.
* Update tutorial to reflect new default arg in Python EndVector.
* Remove optional argument to Python EndVector.
* Add generated files.
* Unset Builder.vectorNumElems when not in use.
* Added missing EndTable() call to VerifyObject()
VerifyObject called VerifyTableStart() but not EndTable(). This made Verifier::VerifyComplexity() increase depth_ with each table, not with the depth of tables.
https://groups.google.com/forum/#!topic/flatbuffers/OpxtW5UFAdg
* Added Check to VerifyAlignment
https://stackoverflow.com/questions/59376308/flatbuffers-verifier-returns-false-without-any-assertion-flatbuffers-debug-veri
* Add GetStringView (Convenience function to get string_view from a String returning an empty string_view on null pointer) like GetString, GetCstring
* flatc should warn, when an attribute is attached more than once.
flatc.exe -b duplicate.fbs
warning: duplicate.fbs(5, 36): warning: attribute already found: priority
duplicate.fbs:
namespace MyGame;
attribute "priority";
table Monster (priority:1, priority:2) {
}
root_type Monster;
* flatc should support --binary --schema with optional scalar fields.
This fixes 'error: Optional scalars are not yet supported in at least one the of the specified programming languages.' when calling flatc.exe --binary --schema with a schema containing optional scalars.
* Generate nullable properties in C# object-based API for optional scalars.
tests\generate_code.bat extended to test this.
Ran tests\generate_code.bat
Ran tests\Flatbuffers.Test\NetTest.bat
* %TEST_BASE_FLAGS% replaced with --gen-object-api in generate_code.bat, because only this is part of this PR. Added this same flag to generate_code.sh
* generate_code.bat and generate_code.sh changed to only test c# with object based api.
This commit makes the names of fuzzing dictionaries the same as the target binary names.
Also it explicitly limits size of test inputs to prevent failures in `regex` and fuzzing time-outs.
* Updated comments and fixed a fundemental type error.
* bump rust flatbuffers semver
* Initial commit with verifier, need to clean up
* Verifier tested. Needs clean up and refactoring.
* Display for InvalidFlatbuffer and better errors for strings
* SimpleToVerify, some refactoring
* Combined VerifierType TableAccessorFuncBody into FollowType
* scrub todos
* Update Rust get_root functions.
There are 6 variants, with verifier options, default verifier options
and no verification "fast".
* Rename root fns
* inline
* Update to use thiserror
* fix for bad compiler
* improve error formatting
* Replace multiply with saturating_multiply
* saturating adds too
* Add docs disclaiming experimental verification system
Co-authored-by: Casper Neo <cneo@google.com>
The flatbuffers::Parser::Parse() isn't an idempotent method for schema parsing.
This commit removes a wrong for-loop that tried to check the same schema twice.
Prior to this commit the default C++ code generator was `c++0x`.
A code generated with `c++0x` code-gen might have a vulnerability (undefined behavior) connected evolution of enums in a schema. This UB could break the backward compatibility if previously generated code casts an unknown enumerator to enum type that knows nothing about future enumerators added to the schema.
The main differences between `c++0x` and `c++11`:
- generated enums use explicitly declared underlying type;
- generated object-API tables don't declare default ctor() explicitly, instead of it default data member initializers are generated.
Please use `flatc --cpp-std c++0x` option for backward compatibility with old compilers.
- add a new method ParseJson to minimize failures during fuzzing
- add default (conditional) move-constructor for Parser
- add a new monster_fuzzer
- switch fuzzers to C++17 and `test/cpp17` generated code
* idl_gen_json_schema.cpp: Changed generation of array element types
#6175
* idl_gen_json_schema.cpp: Simplified indent generation as suggested by @vglavnyy
#6175
* Refactor idl_gen_rust to a ForAllX continuation pattern.
* Removed unneeded SetValue and updated sample rust gencode
* Make Rust flatbuffers print right
* Generated code and removed unnecessary trait constraint
* bumped rust version. Release required
* removed an unwrap in Rust Debug-print unions
* Tested formatting flatbuffers in rust.
* Set float precision in flaky debug-print test
* impl Debug for structs too
Co-authored-by: Casper Neo <cneo@google.com>
* Add test for optional enums in Kotlin
* Rename optional_scalars2.fbs into optional_scalars.fbs
Also updated all references in the project to point to
"optional_scalars.fbs" instead of "optional_scalars2.fbs".
* Rework enums in rust.
They're now a unit struct, rather than an enum. This is a
backwards incompatible change but the previous version had UB
and was also backwards incompatible so...
* Update and test sample rust flatbuffers
* Use bitflags crate to properly support rust enums.
Previously, the bitflags attribute was just ignored. This is a breaking change
as the bitflgs API is not like a normal rust enum (duh).
* variant_name() -> Option<_>
* repr transparent
* Reexport bitflags from flatbuffers
* Make bitflags constants CamelCase, matching normal enums
* Deprecate c-style associated enum constants
Co-authored-by: Casper Neo <cneo@google.com>
* Fixed rust nested flatbuffers for tables other than self
* replaced lifetimes
* Use WrapInNameSpace and also update samples
Co-authored-by: Casper Neo <cneo@google.com>
* GRPC implementation for Typescript
* Fixes a couple of issues
* Finished implementing the typescript support for grpc
* Updated generated code
* Fixes CI
* Moved C++ to optional_scalars2 and added some tests.
Also deleted unused optional_scalars_generated.lobster
* Fixed whitespece in C++ gencode & fixed BUILD file
* Moved C++ onto optional_scalars2 in the .bat file
Co-authored-by: Casper Neo <cneo@google.com>
UnpackTo copies vector elements one-by-one, which can be very inefficient depending on the quality of the compiler optimizations performed. This change updates the operation for vectors of bytes that aren't enums to use 'std::copy', which is usually highly optimized.
vectors of types that are more than one byte can't be optimized in this way because of the endianness of the serialized bytes vs. the target architecture endianness.
vectors of enums can't be optimized because they are required to be static_cast into the appropriate enum type when stored in the vector.
vectors of bools can be optimized in most cases, but since the standard
allows std::vector<bool> template specialization for space-savings,
std::copy doesn't work on every implementation (looking at you
Microsoft). Thus, this optimization is skipped for vector<bool>.
For a specific example, this improves the latency of unpacking large buffers on the Hexagon DSP by about 10x.
Co-authored-by: Matthew Markwell <markwell@google.com>
* [Python] Codegen SizeOf classmethod for structs
This codegens a `SizeOf()` classmethod for all structs since we can't
determine the size of a FlatBuffer generated struct from Python otherwise.
* [JS/TS] Codegen sizeOf static method for structs
This codegens a `sizeOf()` static method for all structs since we can't
determine the size of a FlatBuffer generated struct from JavaScript or
TypeScript otherwise.
* ENABLE_SPAN_T doesn't require UNSAFE_BYTEBUFFER.
Change to ENABLE_SPAN_T that doesn't require UNSAFE_BYTEBUFFER.
* Selectable framework.
Changed target framework to allow selection of 2.0 or 2.1 (or higher)
* Added target framework version check.
* Add core test project.
* Added run on .Net Core.
This optionally generates a static `getFullyQualifiedName()` function to get
the fully-qualified name of a type in JavaScript and TypeScript in a similar
fashion to the C++ codegen.
* Adding FlexBuffers support for Dart language
* Introduce snapshot method.
* Fix docu
* Replacing extension methods with static methods in order to support older Dart version
* Improving code based on PR feedback. Mainly rename refactoring.
* Addressing all PR feedback which does not need clarification
* exchange dynamic type with Object
* Adds better API documentation.
[] operator throws a very descriptive exception in case of a bad key.
* Implementation of JavaScript FlexBuffers decoder
* implements JS FlexBuffers builder
* replacing _toF32 with Math.fround
* Introducing test for BigInt number
* Moving functions from BitWitdth & ValueType object into BitWidthUtil and ValueTypeUtil accordingly.
Removing defensive checks.
Introducing fix for large int numbers by converting them to BigInt type.
Introducing handling for BigInt type in `add` method.
Using TextEncoder and Decoder to handle string / utf8 conversion.
* rename variable
* Lets user turn deduplication strategies for strings, keys and vector of keys off while building FlexBuffer.
Implements quick sort and choses quick sort if the number of keys is bigger then 20.
Removes unnecessary dict lookups in BitWidthUtil helper functions
* make iwidth and uwidth computation simpler and faster
* Making redInt and readUint a bit faster and shim the BigInt / BigUint usage
* Fixing a bug in FlexBuffers JS, where offsets and lengths are stored and read as int and not as uint values.
* Fixing a bug in FlexBuffers Dart, where offset and length values are stored and read as int values instead of uint values
* Adding FlexBuffers support for Dart language
* Introduce snapshot method.
* Fix docu
* Replacing extension methods with static methods in order to support older Dart version
* Improving code based on PR feedback. Mainly rename refactoring.
* Addressing all PR feedback which does not need clarification
* exchange dynamic type with Object
* Adds better API documentation.
[] operator throws a very descriptive exception in case of a bad key.
* Implementation of JavaScript FlexBuffers decoder
* implements JS FlexBuffers builder
* replacing _toF32 with Math.fround
* Introducing test for BigInt number
* Moving functions from BitWitdth & ValueType object into BitWidthUtil and ValueTypeUtil accordingly.
Removing defensive checks.
Introducing fix for large int numbers by converting them to BigInt type.
Introducing handling for BigInt type in `add` method.
Using TextEncoder and Decoder to handle string / utf8 conversion.
* rename variable
* Lets user turn deduplication strategies for strings, keys and vector of keys off while building FlexBuffer.
Implements quick sort and choses quick sort if the number of keys is bigger then 20.
Removes unnecessary dict lookups in BitWidthUtil helper functions
* make iwidth and uwidth computation simpler and faster
* Making redInt and readUint a bit faster and shim the BigInt / BigUint usage
* Cleaned up .NET testing script for Mono
Cleaned up the .NET testing script to make it a little better. It
purposefully doesn't delete the .NET installer and SDk after running the
script, so that they can be used in subsequent invocations. This greatly
speeds up the script.
The downloaded files are ignored by git by default. They can be
explicitly cleaned up by runnning the clean script (clean.sh).
* Trying using older Version indicator
* Remove lins to monsterdata and reference it directly.
* Updated appveryor script to remove copying of files no longer needed
* Continue to update appveyor script to work. Disabled CS0169 in ByteBufferTest
This PR attempts to switch namespace from public enum back to ordinary
concat with _ in Swift. This kept style similar with protobuf, but
different from other popular style guide in Swift.
This is needed because previously, when we do `public enum`, we don't
really know when to declare and when to extend (extension). With namespace
implementation in this PR, there is no such ambiguity.
Adds swift test code
Replaces if statments
Adds swift to supported languages for optionals
Moved std::string to auto
Adds nullable scalars support in object api
* First draft of rust optionals
* Code cleanup around ftBool and ftVectorOfBool
* Tests for Rust optional scalars
* test bools too
Co-authored-by: Casper Neo <cneo@google.com>
This is the first step to upgrade grpc dependency to the latest version.
- Patch protobuf 3.6.1. and grpc 1.15.1 to fix build errors when using the latest Bazel version (3.4.1).
- Add grpc/tests:grpc_test. One can kick off tests in Bazel by calling `bazel test grpc/tests/...`.
- Add missing build targets in tests/BUILD in order to support grpc/tests/BUILD
* Perpares swift to take optional scalars + adds optional string helper method + disables linters in generated code
* Small fix for generated code
* Update grpc support to alpha 17 for swift
* Parser support for nullable scalars
* Use older C++ features
* use default element
* Add a test for json, flexbuffers, and null
* test comments and names
Co-authored-by: Casper Neo <cneo@google.com>
Dart schema compiler generated a static getter for enum values, which
always created a new map instance for its callers. See #5819.
Now it generates const map for better performance
and readability.
Added a method FlatBufferBuilder::createSharedString that
enable string sharing for building messages on java.
The shared pool will only contains strings inserted by
this methods.
Casper