flatbuffers/tests/PythonTest.sh

91 lines
3.4 KiB
Bash
Raw Normal View History

#!/bin/bash -eu
#
# Copyright 2014 Google Inc. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
pushd "$(dirname $0)" >/dev/null
test_dir="$(pwd)"
gen_code_path=${test_dir}
[Python] Python fixed size array (#7529) * feat: Added support for fixed sized arrays to python Problem: We encountered that using fixed arrays from C++ to python that python would not read those arrays correctly due to no size information being encoded in the byte array itself. Fix: Encode the sizes within the generated python file during code generation. Specfically we add GetArrayAsNumpy to the python version of table, which takes as input the length of the vector. When generating the python message files we include this length from the VectorType().fixed_length. * fix: added digit support for camel case to snake case conversion Problem: When including a number in the message name we would encounter cases where SnakeCase would not add the appropirate breaks. e.g. Int32Stamped -> int_32stamped rather than int_32_stamped. Fix: To fix this we can add the condition that we check if the current character is not lower and not a digit, that we check if the previous character was a lower or digit. If it was a lower or digit then we add the break. * fix: Array support for structures Problem: The python generated code for handling non-struct and struct vectors and arrays was inconsistent. The calls to populate the obj api was creating incorrect code. Solution: To fix this the VectorOfStruct and VectorOfNonStruct was rewritten to handle array cases and bring the two methods in line which each other. Testing: PythonTesting.sh now correctly runs and generates the code for array_test.fbs. Minor modifications were done on the test to use the new index accessor for struct arrays and the script correctly sources the location of the python code. * chore: clang format changes * Added code generated by scripts/generate_code. Modified GetArrayOfNonStruct slightly to allow for function overloading allowing the user to get a single element of an array or the whole array. * Added new_line parameter to OffsetPrefix to allow optional new lines to be added. This allows us to use the GenIndents method that automatically adds new lines instead. * Reupload of generated code from the scripts/generate_code.py * Removed new line in GetVectorAsNumpy. * Updated Array lengths to use Length methods where possible. Added fallthrough for GenTypePointer. Added digit check to CamelToSnake method. Added and modified tests for ToSnakeCase and CamelToSnake. * Added range check on the getter methods for vector and array types. Renamed == as is for python
2022-09-22 18:08:09 +00:00
runtime_library_dir=${test_dir}/../../python
# Emit Python code for the example schema in the test dir:
Flatbuffers Python Object API (#5616) * Flatbuffers Python Object API Implement the logic to generate the Python object API that can unpack the data from a buf class into an object class, and pack the data of an object class to a buf class. * Fix the build issues Remove unused parameters and replace auto in the for-loop statement with std::string to make it compatible with VS2010. * Fix the build issues. * Add support for Array type Added logic to handle Array type in Python Object API. Updated the generated code accordingly. * Fix the old style casting from int to char * Fixed another conversion from int to char * Fixed the import for typing Importing typing may cause errors when a machine do not have the moduel typing installed. This PR fixes the issue by guarding "import typing" with the "try/except" statement. * Fix issue of iterating the vector of import list * Update the generated examples using generate_code.sh * Fix the import order for typing The import list was stored in unordered_set, so that each generated codes may have different import order. Therefore, it failed in the consistency test where two generated copies need to have exactly the same apperance. * Optimize unpack using numpy Use numpy to unpack vector whenever it is possible to improve unpack performance. Also, added codegen command for Python specificly in generate_code.sh, because --no-includes cannot be turn on for Python. * Fix the import order * Update generate_code.bat for windows accordingly * Replace error message with pass Avoid printing error message for every Python2 users about typing. Replace it with pass.
2019-12-02 22:11:28 +00:00
${test_dir}/../flatc -p -o ${gen_code_path} -I include_test monster_test.fbs --gen-object-api
${test_dir}/../flatc -p -o ${gen_code_path} -I include_test monster_test.fbs --gen-object-api --gen-onefile
${test_dir}/../flatc -p -o ${gen_code_path} -I include_test monster_extra.fbs --gen-object-api
[Python] Python fixed size array (#7529) * feat: Added support for fixed sized arrays to python Problem: We encountered that using fixed arrays from C++ to python that python would not read those arrays correctly due to no size information being encoded in the byte array itself. Fix: Encode the sizes within the generated python file during code generation. Specfically we add GetArrayAsNumpy to the python version of table, which takes as input the length of the vector. When generating the python message files we include this length from the VectorType().fixed_length. * fix: added digit support for camel case to snake case conversion Problem: When including a number in the message name we would encounter cases where SnakeCase would not add the appropirate breaks. e.g. Int32Stamped -> int_32stamped rather than int_32_stamped. Fix: To fix this we can add the condition that we check if the current character is not lower and not a digit, that we check if the previous character was a lower or digit. If it was a lower or digit then we add the break. * fix: Array support for structures Problem: The python generated code for handling non-struct and struct vectors and arrays was inconsistent. The calls to populate the obj api was creating incorrect code. Solution: To fix this the VectorOfStruct and VectorOfNonStruct was rewritten to handle array cases and bring the two methods in line which each other. Testing: PythonTesting.sh now correctly runs and generates the code for array_test.fbs. Minor modifications were done on the test to use the new index accessor for struct arrays and the script correctly sources the location of the python code. * chore: clang format changes * Added code generated by scripts/generate_code. Modified GetArrayOfNonStruct slightly to allow for function overloading allowing the user to get a single element of an array or the whole array. * Added new_line parameter to OffsetPrefix to allow optional new lines to be added. This allows us to use the GenIndents method that automatically adds new lines instead. * Reupload of generated code from the scripts/generate_code.py * Removed new line in GetVectorAsNumpy. * Updated Array lengths to use Length methods where possible. Added fallthrough for GenTypePointer. Added digit check to CamelToSnake method. Added and modified tests for ToSnakeCase and CamelToSnake. * Added range check on the getter methods for vector and array types. Renamed == as is for python
2022-09-22 18:08:09 +00:00
${test_dir}/../flatc -p -o ${gen_code_path} -I include_test arrays_test.fbs --gen-object-api
${test_dir}/../flatc -p -o ${gen_code_path} -I include_test nested_union_test.fbs --gen-object-api
# Syntax: run_tests <interpreter> <benchmark vtable dedupes>
# <benchmark read count> <benchmark build count>
interpreters_tested=()
function run_tests() {
if $(which ${1} >/dev/null); then
echo "Testing with interpreter: ${1}"
PYTHONDONTWRITEBYTECODE=1 \
JYTHONDONTWRITEBYTECODE=1 \
PYTHONPATH=${runtime_library_dir}:${gen_code_path} \
JYTHONPATH=${runtime_library_dir}:${gen_code_path} \
COMPARE_GENERATED_TO_GO=0 \
COMPARE_GENERATED_TO_JAVA=0 \
$1 py_test.py $2 $3 $4 $5
if [ $1 = python3 ]; then
PYTHONDONTWRITEBYTECODE=1 \
PYTHONPATH=${runtime_library_dir}:${gen_code_path} \
$1 py_flexbuffers_test.py
fi
interpreters_tested+=(${1})
echo
fi
}
# Run test suite with these interpreters. The arguments are benchmark counts.
run_tests python2.6 100 100 100 false
run_tests python2.7 100 100 100 false
run_tests python2.7 100 100 100 true
run_tests python3 100 100 100 false
run_tests python3 100 100 100 true
run_tests pypy 100 100 100 false
# NOTE: We'd like to support python2.5 in the future.
# NOTE: Jython 2.7.0 fails due to a bug in the stdlib `struct` library:
# http://bugs.jython.org/issue2188
if [ ${#interpreters_tested[@]} -eq 0 ]; then
echo "No Python interpeters found on this system, could not run tests."
exit 1
fi
# Run test suite with default python intereter.
# (If the Python program `coverage` is available, it will be run, too.
# Install `coverage` with `pip install coverage`.)
if $(which coverage >/dev/null); then
echo 'Found coverage utility, running coverage with default Python:'
PYTHONDONTWRITEBYTECODE=1 \
PYTHONPATH=${runtime_library_dir}:${gen_code_path} \
[Python] Python fixed size array (#7529) * feat: Added support for fixed sized arrays to python Problem: We encountered that using fixed arrays from C++ to python that python would not read those arrays correctly due to no size information being encoded in the byte array itself. Fix: Encode the sizes within the generated python file during code generation. Specfically we add GetArrayAsNumpy to the python version of table, which takes as input the length of the vector. When generating the python message files we include this length from the VectorType().fixed_length. * fix: added digit support for camel case to snake case conversion Problem: When including a number in the message name we would encounter cases where SnakeCase would not add the appropirate breaks. e.g. Int32Stamped -> int_32stamped rather than int_32_stamped. Fix: To fix this we can add the condition that we check if the current character is not lower and not a digit, that we check if the previous character was a lower or digit. If it was a lower or digit then we add the break. * fix: Array support for structures Problem: The python generated code for handling non-struct and struct vectors and arrays was inconsistent. The calls to populate the obj api was creating incorrect code. Solution: To fix this the VectorOfStruct and VectorOfNonStruct was rewritten to handle array cases and bring the two methods in line which each other. Testing: PythonTesting.sh now correctly runs and generates the code for array_test.fbs. Minor modifications were done on the test to use the new index accessor for struct arrays and the script correctly sources the location of the python code. * chore: clang format changes * Added code generated by scripts/generate_code. Modified GetArrayOfNonStruct slightly to allow for function overloading allowing the user to get a single element of an array or the whole array. * Added new_line parameter to OffsetPrefix to allow optional new lines to be added. This allows us to use the GenIndents method that automatically adds new lines instead. * Reupload of generated code from the scripts/generate_code.py * Removed new line in GetVectorAsNumpy. * Updated Array lengths to use Length methods where possible. Added fallthrough for GenTypePointer. Added digit check to CamelToSnake method. Added and modified tests for ToSnakeCase and CamelToSnake. * Added range check on the getter methods for vector and array types. Renamed == as is for python
2022-09-22 18:08:09 +00:00
coverage run --source=flatbuffers,MyGame py_test.py 0 0 0 false > /dev/null
echo
cov_result=`coverage report --omit="*flatbuffers/vendor*,*py_test*" \
| tail -n 1 | awk ' { print $4 } '`
echo "Code coverage: ${cov_result}"
else
echo -n "Did not find coverage utility for default Python, skipping. "
echo "Install with 'pip install coverage'."
fi
echo
echo "OK: all tests passed for ${#interpreters_tested[@]} interpreters: ${interpreters_tested[@]}."