flatbuffers/build_defs.bzl

279 lines
10 KiB
Python
Raw Normal View History

# Description:
# BUILD rules for generating flatbuffer files in various languages.
"""
Rules for building C++ flatbuffers with Bazel.
"""
load("@rules_cc//cc:defs.bzl", "cc_library")
[TS/JS] Entry point per namespace and reworked 1.x compatible single file build (#7510) * [TS/JS] Entry point per namespace * Fix handling of outputpath and array_test * Attempt to fix generate_code * Fix cwd for ts in generate_code * Attempt to fixup bazel and some docs * Add --ts-flat-files to bazel build to get bundle * Move to DEFAULT_FLATC_TS_ARGS * Attempt to add esbuild * Attempt to use npm instead * Remove futile attempt to add esbuild * Attempt to as bazel esbuild * Shuffle * Upgrade bazel deps * Revert failed attempts to get bazel working * Ignore flatc tests for now * Add esbuild dependency * `package.json` Include esbuild * `WORKSPACE` Add fetching esbuild binary * Update WORKSPACE * Unfreeze Lockfile * Update WORKSPACE * Update BUILD.bazel * Rework to suggest instead of running external bundler * Add esbuild generation to test script * Prelim bundle test * Run test JavaScriptTest from flatbuffers 1.x * Deps upgrade * Clang format fix * Revert bazel changes * Fix newline * Generate with type declarations * Handle "empty" root namespace * Adjust tests for typescript_keywords.ts * Separate test procedure for old node resolution module output * Fix rel path for root level re-exports * Bazel support for esbuild-based flatc Unfortunately, we lose typing information because the new esbuild method of generating single files does not generate type information. The method used here is a bit hack-ish because it relies on parsing the console output of flatc to figure out what to do. * Try to fix bazel build for when node isn't present on host * Auto formatting fixes * Fix missing generated code Co-authored-by: Derek Bailey <derekbailey@google.com> Co-authored-by: James Kuszmaul <jabukuszmaul+collab@gmail.com>
2023-01-21 20:22:22 +00:00
TRUE_FLATC_PATH = "@com_github_google_flatbuffers//:flatc"
DEFAULT_INCLUDE_PATHS = [
"./",
"$(GENDIR)",
"$(BINDIR)",
"$(execpath @com_github_google_flatbuffers//:flatc).runfiles/com_github_google_flatbuffers",
]
[TS/JS] Entry point per namespace and reworked 1.x compatible single file build (#7510) * [TS/JS] Entry point per namespace * Fix handling of outputpath and array_test * Attempt to fix generate_code * Fix cwd for ts in generate_code * Attempt to fixup bazel and some docs * Add --ts-flat-files to bazel build to get bundle * Move to DEFAULT_FLATC_TS_ARGS * Attempt to add esbuild * Attempt to use npm instead * Remove futile attempt to add esbuild * Attempt to as bazel esbuild * Shuffle * Upgrade bazel deps * Revert failed attempts to get bazel working * Ignore flatc tests for now * Add esbuild dependency * `package.json` Include esbuild * `WORKSPACE` Add fetching esbuild binary * Update WORKSPACE * Unfreeze Lockfile * Update WORKSPACE * Update BUILD.bazel * Rework to suggest instead of running external bundler * Add esbuild generation to test script * Prelim bundle test * Run test JavaScriptTest from flatbuffers 1.x * Deps upgrade * Clang format fix * Revert bazel changes * Fix newline * Generate with type declarations * Handle "empty" root namespace * Adjust tests for typescript_keywords.ts * Separate test procedure for old node resolution module output * Fix rel path for root level re-exports * Bazel support for esbuild-based flatc Unfortunately, we lose typing information because the new esbuild method of generating single files does not generate type information. The method used here is a bit hack-ish because it relies on parsing the console output of flatc to figure out what to do. * Try to fix bazel build for when node isn't present on host * Auto formatting fixes * Fix missing generated code Co-authored-by: Derek Bailey <derekbailey@google.com> Co-authored-by: James Kuszmaul <jabukuszmaul+collab@gmail.com>
2023-01-21 20:22:22 +00:00
def default_include_paths(flatc_path):
return [
"./",
"$(GENDIR)",
"$(BINDIR)",
"$(execpath %s).runfiles/com_github_google_flatbuffers" % (flatc_path),
]
DEFAULT_FLATC_ARGS = [
"--gen-object-api",
"--gen-compare",
"--no-includes",
"--gen-mutable",
"--reflect-names",
"--cpp-ptr-type flatbuffers::unique_ptr",
]
def flatbuffer_library_public(
name,
srcs,
outs,
language_flag,
out_prefix = "",
includes = [],
[TS/JS] Entry point per namespace and reworked 1.x compatible single file build (#7510) * [TS/JS] Entry point per namespace * Fix handling of outputpath and array_test * Attempt to fix generate_code * Fix cwd for ts in generate_code * Attempt to fixup bazel and some docs * Add --ts-flat-files to bazel build to get bundle * Move to DEFAULT_FLATC_TS_ARGS * Attempt to add esbuild * Attempt to use npm instead * Remove futile attempt to add esbuild * Attempt to as bazel esbuild * Shuffle * Upgrade bazel deps * Revert failed attempts to get bazel working * Ignore flatc tests for now * Add esbuild dependency * `package.json` Include esbuild * `WORKSPACE` Add fetching esbuild binary * Update WORKSPACE * Unfreeze Lockfile * Update WORKSPACE * Update BUILD.bazel * Rework to suggest instead of running external bundler * Add esbuild generation to test script * Prelim bundle test * Run test JavaScriptTest from flatbuffers 1.x * Deps upgrade * Clang format fix * Revert bazel changes * Fix newline * Generate with type declarations * Handle "empty" root namespace * Adjust tests for typescript_keywords.ts * Separate test procedure for old node resolution module output * Fix rel path for root level re-exports * Bazel support for esbuild-based flatc Unfortunately, we lose typing information because the new esbuild method of generating single files does not generate type information. The method used here is a bit hack-ish because it relies on parsing the console output of flatc to figure out what to do. * Try to fix bazel build for when node isn't present on host * Auto formatting fixes * Fix missing generated code Co-authored-by: Derek Bailey <derekbailey@google.com> Co-authored-by: James Kuszmaul <jabukuszmaul+collab@gmail.com>
2023-01-21 20:22:22 +00:00
include_paths = None,
flatc_args = DEFAULT_FLATC_ARGS,
reflection_name = "",
2019-05-06 19:10:11 +00:00
reflection_visibility = None,
compatible_with = None,
restricted_to = None,
target_compatible_with = None,
[TS/JS] Entry point per namespace and reworked 1.x compatible single file build (#7510) * [TS/JS] Entry point per namespace * Fix handling of outputpath and array_test * Attempt to fix generate_code * Fix cwd for ts in generate_code * Attempt to fixup bazel and some docs * Add --ts-flat-files to bazel build to get bundle * Move to DEFAULT_FLATC_TS_ARGS * Attempt to add esbuild * Attempt to use npm instead * Remove futile attempt to add esbuild * Attempt to as bazel esbuild * Shuffle * Upgrade bazel deps * Revert failed attempts to get bazel working * Ignore flatc tests for now * Add esbuild dependency * `package.json` Include esbuild * `WORKSPACE` Add fetching esbuild binary * Update WORKSPACE * Unfreeze Lockfile * Update WORKSPACE * Update BUILD.bazel * Rework to suggest instead of running external bundler * Add esbuild generation to test script * Prelim bundle test * Run test JavaScriptTest from flatbuffers 1.x * Deps upgrade * Clang format fix * Revert bazel changes * Fix newline * Generate with type declarations * Handle "empty" root namespace * Adjust tests for typescript_keywords.ts * Separate test procedure for old node resolution module output * Fix rel path for root level re-exports * Bazel support for esbuild-based flatc Unfortunately, we lose typing information because the new esbuild method of generating single files does not generate type information. The method used here is a bit hack-ish because it relies on parsing the console output of flatc to figure out what to do. * Try to fix bazel build for when node isn't present on host * Auto formatting fixes * Fix missing generated code Co-authored-by: Derek Bailey <derekbailey@google.com> Co-authored-by: James Kuszmaul <jabukuszmaul+collab@gmail.com>
2023-01-21 20:22:22 +00:00
flatc_path = "@com_github_google_flatbuffers//:flatc",
Migrate from rules_nodejs to rules_js/rules_ts (take 2) (#7928) * Migrate from rules_nodejs to rules_js/rules_ts (take 2) This is the second version of patch #7923. The first version got reverted because bazel query was failing: $ bazel --nosystem_rc --nohome_rc query tests(set('//...')) except tests(attr("tags", "manual", set('//...'))) ERROR: Traceback (most recent call last): File "/workdir/tests/ts/bazel_repository_test_dir/BUILD", line 6, column 22, in <toplevel> npm_link_all_packages(name = "node_modules") File "/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/ec321eb2cc2d0f8f91b676b6d4c66c29/external/npm/defs.bzl", line 188, column 13, in npm_link_all_packages fail(msg) Error in fail: The npm_link_all_packages() macro loaded from @npm//:defs.bzl and called in bazel package 'tests/ts/bazel_repository_test_dir' may only be called in bazel packages that correspond to the pnpm root package '' and pnpm workspace projects '' This was happening because the `.bazelrc` file only added `--deleted_packages` to the `build` command. We also need it for the `query` command. This second version of the patch fixes that. Original commit message: This patch migrates the current use of rules_nodejs to the new rules_js. rules_js is the intended replacement of rules_nodejs as per this note: https://github.com/aspect-build/rules_js#relationship-to-rules_nodejs > rules_js is an alternative to the build_bazel_rules_nodejs Bazel module > and accompanying npm packages hosted in > https://github.com/bazelbuild/rules_nodejs, which is now > unmaintained. All users are recommended to use rules_js instead. There are a few notable changes in this patch: 1. The `flatbuffer_ts_library` macro no longer accepts a `package_name` attribute. This is because rules_js appears to manage the import naming of dependencies via top-level `npm_link_package` targets. Users will have to migrate. 2. I added a few more arguments to `flatbuffer_library_public()`. These helped with exposing esbuild to `ts/compile_flat_file.sh`. 3. I pinned the version of `typescript` in `package.json` so that rules_ts can download the exact same version. rules_ts doesn't know what to do if the version isn't exact. 4. Since rules_js uses the pnpm locking mechanism, we now have a `pnpm-lock.yaml` file instead of a yarn lock file. 4. I added bazel targets for a few of the existing tests in `tests/ts`. They can be run with `bazel test //test/ts:all`. Since there is no flexbuffers bazel target, I did not add a bazel target for the corresponding test. 5. I added a separate workspace in `tests/ts/bazel_repository_test_dir/` to validate that the flatbuffers code can be imported as an external repository. You can run the test with `bazel test //test/ts:bazel_repository_test`. For this to work, I needed to expose a non-trivial chunk of the flatbuffers code to the test. I achieved this through some recursive `distribution` filegroups. This is inspired by rules_python's workspace tests. I did not do anything special to validate that the `gen_reflections` parameter works the same. This patch doesn't change anything about the TypeScript generation. As a side note: I am not an expert with rules_js. This patch is my attempt based on my limited understanding of the rule set. Fixes #7817 * Fix the query --------- Co-authored-by: Derek Bailey <derekbailey@google.com>
2023-05-03 18:48:15 +00:00
output_to_bindir = False,
tools = None,
extra_env = None,
**kwargs):
"""Generates code files for reading/writing the given flatbuffers in the requested language using the public compiler.
Args:
name: Rule name.
srcs: Source .fbs files. Sent in order to the compiler.
outs: Output files from flatc.
language_flag: Target language flag. One of [-c, -j, -js].
out_prefix: Prepend this path to the front of all generated files except on
single source targets. Usually is a directory name.
includes: Optional, list of filegroups of schemas that the srcs depend on.
include_paths: Optional, list of paths the includes files can be found in.
flatc_args: Optional, list of additional arguments to pass to flatc.
reflection_name: Optional, if set this will generate the flatbuffer
reflection binaries for the schemas.
2019-05-06 19:10:11 +00:00
reflection_visibility: The visibility of the generated reflection Fileset.
output_to_bindir: Passed to genrule for output to bin directory.
compatible_with: Optional, The list of environments this rule can be
built for, in addition to default-supported environments.
restricted_to: Optional, The list of environments this rule can be built
for, instead of default-supported environments.
target_compatible_with: Optional, The list of target platform constraints
to use.
[TS/JS] Entry point per namespace and reworked 1.x compatible single file build (#7510) * [TS/JS] Entry point per namespace * Fix handling of outputpath and array_test * Attempt to fix generate_code * Fix cwd for ts in generate_code * Attempt to fixup bazel and some docs * Add --ts-flat-files to bazel build to get bundle * Move to DEFAULT_FLATC_TS_ARGS * Attempt to add esbuild * Attempt to use npm instead * Remove futile attempt to add esbuild * Attempt to as bazel esbuild * Shuffle * Upgrade bazel deps * Revert failed attempts to get bazel working * Ignore flatc tests for now * Add esbuild dependency * `package.json` Include esbuild * `WORKSPACE` Add fetching esbuild binary * Update WORKSPACE * Unfreeze Lockfile * Update WORKSPACE * Update BUILD.bazel * Rework to suggest instead of running external bundler * Add esbuild generation to test script * Prelim bundle test * Run test JavaScriptTest from flatbuffers 1.x * Deps upgrade * Clang format fix * Revert bazel changes * Fix newline * Generate with type declarations * Handle "empty" root namespace * Adjust tests for typescript_keywords.ts * Separate test procedure for old node resolution module output * Fix rel path for root level re-exports * Bazel support for esbuild-based flatc Unfortunately, we lose typing information because the new esbuild method of generating single files does not generate type information. The method used here is a bit hack-ish because it relies on parsing the console output of flatc to figure out what to do. * Try to fix bazel build for when node isn't present on host * Auto formatting fixes * Fix missing generated code Co-authored-by: Derek Bailey <derekbailey@google.com> Co-authored-by: James Kuszmaul <jabukuszmaul+collab@gmail.com>
2023-01-21 20:22:22 +00:00
flatc_path: Bazel target corresponding to the flatc compiler to use.
output_to_bindir: Passed to genrule for output to bin directory.
Migrate from rules_nodejs to rules_js/rules_ts (take 2) (#7928) * Migrate from rules_nodejs to rules_js/rules_ts (take 2) This is the second version of patch #7923. The first version got reverted because bazel query was failing: $ bazel --nosystem_rc --nohome_rc query tests(set('//...')) except tests(attr("tags", "manual", set('//...'))) ERROR: Traceback (most recent call last): File "/workdir/tests/ts/bazel_repository_test_dir/BUILD", line 6, column 22, in <toplevel> npm_link_all_packages(name = "node_modules") File "/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/ec321eb2cc2d0f8f91b676b6d4c66c29/external/npm/defs.bzl", line 188, column 13, in npm_link_all_packages fail(msg) Error in fail: The npm_link_all_packages() macro loaded from @npm//:defs.bzl and called in bazel package 'tests/ts/bazel_repository_test_dir' may only be called in bazel packages that correspond to the pnpm root package '' and pnpm workspace projects '' This was happening because the `.bazelrc` file only added `--deleted_packages` to the `build` command. We also need it for the `query` command. This second version of the patch fixes that. Original commit message: This patch migrates the current use of rules_nodejs to the new rules_js. rules_js is the intended replacement of rules_nodejs as per this note: https://github.com/aspect-build/rules_js#relationship-to-rules_nodejs > rules_js is an alternative to the build_bazel_rules_nodejs Bazel module > and accompanying npm packages hosted in > https://github.com/bazelbuild/rules_nodejs, which is now > unmaintained. All users are recommended to use rules_js instead. There are a few notable changes in this patch: 1. The `flatbuffer_ts_library` macro no longer accepts a `package_name` attribute. This is because rules_js appears to manage the import naming of dependencies via top-level `npm_link_package` targets. Users will have to migrate. 2. I added a few more arguments to `flatbuffer_library_public()`. These helped with exposing esbuild to `ts/compile_flat_file.sh`. 3. I pinned the version of `typescript` in `package.json` so that rules_ts can download the exact same version. rules_ts doesn't know what to do if the version isn't exact. 4. Since rules_js uses the pnpm locking mechanism, we now have a `pnpm-lock.yaml` file instead of a yarn lock file. 4. I added bazel targets for a few of the existing tests in `tests/ts`. They can be run with `bazel test //test/ts:all`. Since there is no flexbuffers bazel target, I did not add a bazel target for the corresponding test. 5. I added a separate workspace in `tests/ts/bazel_repository_test_dir/` to validate that the flatbuffers code can be imported as an external repository. You can run the test with `bazel test //test/ts:bazel_repository_test`. For this to work, I needed to expose a non-trivial chunk of the flatbuffers code to the test. I achieved this through some recursive `distribution` filegroups. This is inspired by rules_python's workspace tests. I did not do anything special to validate that the `gen_reflections` parameter works the same. This patch doesn't change anything about the TypeScript generation. As a side note: I am not an expert with rules_js. This patch is my attempt based on my limited understanding of the rule set. Fixes #7817 * Fix the query --------- Co-authored-by: Derek Bailey <derekbailey@google.com>
2023-05-03 18:48:15 +00:00
tools: Optional, passed to genrule for list of tools to make available
during the action.
extra_env: Optional, must be a string of "VAR1=VAL1 VAR2=VAL2". These get
set as environment variables that "flatc_path" sees.
**kwargs: Passed to the underlying genrule.
This rule creates a filegroup(name) with all generated source files, and
optionally a Fileset([reflection_name]) with all generated reflection
binaries.
"""
[TS/JS] Entry point per namespace and reworked 1.x compatible single file build (#7510) * [TS/JS] Entry point per namespace * Fix handling of outputpath and array_test * Attempt to fix generate_code * Fix cwd for ts in generate_code * Attempt to fixup bazel and some docs * Add --ts-flat-files to bazel build to get bundle * Move to DEFAULT_FLATC_TS_ARGS * Attempt to add esbuild * Attempt to use npm instead * Remove futile attempt to add esbuild * Attempt to as bazel esbuild * Shuffle * Upgrade bazel deps * Revert failed attempts to get bazel working * Ignore flatc tests for now * Add esbuild dependency * `package.json` Include esbuild * `WORKSPACE` Add fetching esbuild binary * Update WORKSPACE * Unfreeze Lockfile * Update WORKSPACE * Update BUILD.bazel * Rework to suggest instead of running external bundler * Add esbuild generation to test script * Prelim bundle test * Run test JavaScriptTest from flatbuffers 1.x * Deps upgrade * Clang format fix * Revert bazel changes * Fix newline * Generate with type declarations * Handle "empty" root namespace * Adjust tests for typescript_keywords.ts * Separate test procedure for old node resolution module output * Fix rel path for root level re-exports * Bazel support for esbuild-based flatc Unfortunately, we lose typing information because the new esbuild method of generating single files does not generate type information. The method used here is a bit hack-ish because it relies on parsing the console output of flatc to figure out what to do. * Try to fix bazel build for when node isn't present on host * Auto formatting fixes * Fix missing generated code Co-authored-by: Derek Bailey <derekbailey@google.com> Co-authored-by: James Kuszmaul <jabukuszmaul+collab@gmail.com>
2023-01-21 20:22:22 +00:00
if include_paths == None:
include_paths = default_include_paths(flatc_path)
include_paths_cmd = ["-I %s" % (s) for s in include_paths]
Migrate from rules_nodejs to rules_js/rules_ts (take 2) (#7928) * Migrate from rules_nodejs to rules_js/rules_ts (take 2) This is the second version of patch #7923. The first version got reverted because bazel query was failing: $ bazel --nosystem_rc --nohome_rc query tests(set('//...')) except tests(attr("tags", "manual", set('//...'))) ERROR: Traceback (most recent call last): File "/workdir/tests/ts/bazel_repository_test_dir/BUILD", line 6, column 22, in <toplevel> npm_link_all_packages(name = "node_modules") File "/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/ec321eb2cc2d0f8f91b676b6d4c66c29/external/npm/defs.bzl", line 188, column 13, in npm_link_all_packages fail(msg) Error in fail: The npm_link_all_packages() macro loaded from @npm//:defs.bzl and called in bazel package 'tests/ts/bazel_repository_test_dir' may only be called in bazel packages that correspond to the pnpm root package '' and pnpm workspace projects '' This was happening because the `.bazelrc` file only added `--deleted_packages` to the `build` command. We also need it for the `query` command. This second version of the patch fixes that. Original commit message: This patch migrates the current use of rules_nodejs to the new rules_js. rules_js is the intended replacement of rules_nodejs as per this note: https://github.com/aspect-build/rules_js#relationship-to-rules_nodejs > rules_js is an alternative to the build_bazel_rules_nodejs Bazel module > and accompanying npm packages hosted in > https://github.com/bazelbuild/rules_nodejs, which is now > unmaintained. All users are recommended to use rules_js instead. There are a few notable changes in this patch: 1. The `flatbuffer_ts_library` macro no longer accepts a `package_name` attribute. This is because rules_js appears to manage the import naming of dependencies via top-level `npm_link_package` targets. Users will have to migrate. 2. I added a few more arguments to `flatbuffer_library_public()`. These helped with exposing esbuild to `ts/compile_flat_file.sh`. 3. I pinned the version of `typescript` in `package.json` so that rules_ts can download the exact same version. rules_ts doesn't know what to do if the version isn't exact. 4. Since rules_js uses the pnpm locking mechanism, we now have a `pnpm-lock.yaml` file instead of a yarn lock file. 4. I added bazel targets for a few of the existing tests in `tests/ts`. They can be run with `bazel test //test/ts:all`. Since there is no flexbuffers bazel target, I did not add a bazel target for the corresponding test. 5. I added a separate workspace in `tests/ts/bazel_repository_test_dir/` to validate that the flatbuffers code can be imported as an external repository. You can run the test with `bazel test //test/ts:bazel_repository_test`. For this to work, I needed to expose a non-trivial chunk of the flatbuffers code to the test. I achieved this through some recursive `distribution` filegroups. This is inspired by rules_python's workspace tests. I did not do anything special to validate that the `gen_reflections` parameter works the same. This patch doesn't change anything about the TypeScript generation. As a side note: I am not an expert with rules_js. This patch is my attempt based on my limited understanding of the rule set. Fixes #7817 * Fix the query --------- Co-authored-by: Derek Bailey <derekbailey@google.com>
2023-05-03 18:48:15 +00:00
extra_env = extra_env or ""
# '$(@D)' when given a single source target will give the appropriate
# directory. Appending 'out_prefix' is only necessary when given a build
# target with multiple sources.
output_directory = (
("-o $(@D)/%s" % (out_prefix)) if len(srcs) > 1 else ("-o $(@D)")
)
genrule_cmd = " ".join([
"SRCS=($(SRCS));",
"for f in $${SRCS[@]:0:%s}; do" % len(srcs),
Migrate from rules_nodejs to rules_js/rules_ts (take 2) (#7928) * Migrate from rules_nodejs to rules_js/rules_ts (take 2) This is the second version of patch #7923. The first version got reverted because bazel query was failing: $ bazel --nosystem_rc --nohome_rc query tests(set('//...')) except tests(attr("tags", "manual", set('//...'))) ERROR: Traceback (most recent call last): File "/workdir/tests/ts/bazel_repository_test_dir/BUILD", line 6, column 22, in <toplevel> npm_link_all_packages(name = "node_modules") File "/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/ec321eb2cc2d0f8f91b676b6d4c66c29/external/npm/defs.bzl", line 188, column 13, in npm_link_all_packages fail(msg) Error in fail: The npm_link_all_packages() macro loaded from @npm//:defs.bzl and called in bazel package 'tests/ts/bazel_repository_test_dir' may only be called in bazel packages that correspond to the pnpm root package '' and pnpm workspace projects '' This was happening because the `.bazelrc` file only added `--deleted_packages` to the `build` command. We also need it for the `query` command. This second version of the patch fixes that. Original commit message: This patch migrates the current use of rules_nodejs to the new rules_js. rules_js is the intended replacement of rules_nodejs as per this note: https://github.com/aspect-build/rules_js#relationship-to-rules_nodejs > rules_js is an alternative to the build_bazel_rules_nodejs Bazel module > and accompanying npm packages hosted in > https://github.com/bazelbuild/rules_nodejs, which is now > unmaintained. All users are recommended to use rules_js instead. There are a few notable changes in this patch: 1. The `flatbuffer_ts_library` macro no longer accepts a `package_name` attribute. This is because rules_js appears to manage the import naming of dependencies via top-level `npm_link_package` targets. Users will have to migrate. 2. I added a few more arguments to `flatbuffer_library_public()`. These helped with exposing esbuild to `ts/compile_flat_file.sh`. 3. I pinned the version of `typescript` in `package.json` so that rules_ts can download the exact same version. rules_ts doesn't know what to do if the version isn't exact. 4. Since rules_js uses the pnpm locking mechanism, we now have a `pnpm-lock.yaml` file instead of a yarn lock file. 4. I added bazel targets for a few of the existing tests in `tests/ts`. They can be run with `bazel test //test/ts:all`. Since there is no flexbuffers bazel target, I did not add a bazel target for the corresponding test. 5. I added a separate workspace in `tests/ts/bazel_repository_test_dir/` to validate that the flatbuffers code can be imported as an external repository. You can run the test with `bazel test //test/ts:bazel_repository_test`. For this to work, I needed to expose a non-trivial chunk of the flatbuffers code to the test. I achieved this through some recursive `distribution` filegroups. This is inspired by rules_python's workspace tests. I did not do anything special to validate that the `gen_reflections` parameter works the same. This patch doesn't change anything about the TypeScript generation. As a side note: I am not an expert with rules_js. This patch is my attempt based on my limited understanding of the rule set. Fixes #7817 * Fix the query --------- Co-authored-by: Derek Bailey <derekbailey@google.com>
2023-05-03 18:48:15 +00:00
"OUTPUT_FILE=\"$(OUTS)\" %s $(location %s)" % (extra_env, flatc_path),
" ".join(include_paths_cmd),
" ".join(flatc_args),
language_flag,
output_directory,
"$$f;",
"done",
])
native.genrule(
name = name,
srcs = srcs + includes,
outs = outs,
output_to_bindir = output_to_bindir,
Migrate from rules_nodejs to rules_js/rules_ts (take 2) (#7928) * Migrate from rules_nodejs to rules_js/rules_ts (take 2) This is the second version of patch #7923. The first version got reverted because bazel query was failing: $ bazel --nosystem_rc --nohome_rc query tests(set('//...')) except tests(attr("tags", "manual", set('//...'))) ERROR: Traceback (most recent call last): File "/workdir/tests/ts/bazel_repository_test_dir/BUILD", line 6, column 22, in <toplevel> npm_link_all_packages(name = "node_modules") File "/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/ec321eb2cc2d0f8f91b676b6d4c66c29/external/npm/defs.bzl", line 188, column 13, in npm_link_all_packages fail(msg) Error in fail: The npm_link_all_packages() macro loaded from @npm//:defs.bzl and called in bazel package 'tests/ts/bazel_repository_test_dir' may only be called in bazel packages that correspond to the pnpm root package '' and pnpm workspace projects '' This was happening because the `.bazelrc` file only added `--deleted_packages` to the `build` command. We also need it for the `query` command. This second version of the patch fixes that. Original commit message: This patch migrates the current use of rules_nodejs to the new rules_js. rules_js is the intended replacement of rules_nodejs as per this note: https://github.com/aspect-build/rules_js#relationship-to-rules_nodejs > rules_js is an alternative to the build_bazel_rules_nodejs Bazel module > and accompanying npm packages hosted in > https://github.com/bazelbuild/rules_nodejs, which is now > unmaintained. All users are recommended to use rules_js instead. There are a few notable changes in this patch: 1. The `flatbuffer_ts_library` macro no longer accepts a `package_name` attribute. This is because rules_js appears to manage the import naming of dependencies via top-level `npm_link_package` targets. Users will have to migrate. 2. I added a few more arguments to `flatbuffer_library_public()`. These helped with exposing esbuild to `ts/compile_flat_file.sh`. 3. I pinned the version of `typescript` in `package.json` so that rules_ts can download the exact same version. rules_ts doesn't know what to do if the version isn't exact. 4. Since rules_js uses the pnpm locking mechanism, we now have a `pnpm-lock.yaml` file instead of a yarn lock file. 4. I added bazel targets for a few of the existing tests in `tests/ts`. They can be run with `bazel test //test/ts:all`. Since there is no flexbuffers bazel target, I did not add a bazel target for the corresponding test. 5. I added a separate workspace in `tests/ts/bazel_repository_test_dir/` to validate that the flatbuffers code can be imported as an external repository. You can run the test with `bazel test //test/ts:bazel_repository_test`. For this to work, I needed to expose a non-trivial chunk of the flatbuffers code to the test. I achieved this through some recursive `distribution` filegroups. This is inspired by rules_python's workspace tests. I did not do anything special to validate that the `gen_reflections` parameter works the same. This patch doesn't change anything about the TypeScript generation. As a side note: I am not an expert with rules_js. This patch is my attempt based on my limited understanding of the rule set. Fixes #7817 * Fix the query --------- Co-authored-by: Derek Bailey <derekbailey@google.com>
2023-05-03 18:48:15 +00:00
tools = (tools or []) + [flatc_path],
cmd = genrule_cmd,
compatible_with = compatible_with,
target_compatible_with = target_compatible_with,
restricted_to = restricted_to,
message = "Generating flatbuffer files for %s:" % (name),
Migrate from rules_nodejs to rules_js/rules_ts (take 2) (#7928) * Migrate from rules_nodejs to rules_js/rules_ts (take 2) This is the second version of patch #7923. The first version got reverted because bazel query was failing: $ bazel --nosystem_rc --nohome_rc query tests(set('//...')) except tests(attr("tags", "manual", set('//...'))) ERROR: Traceback (most recent call last): File "/workdir/tests/ts/bazel_repository_test_dir/BUILD", line 6, column 22, in <toplevel> npm_link_all_packages(name = "node_modules") File "/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/ec321eb2cc2d0f8f91b676b6d4c66c29/external/npm/defs.bzl", line 188, column 13, in npm_link_all_packages fail(msg) Error in fail: The npm_link_all_packages() macro loaded from @npm//:defs.bzl and called in bazel package 'tests/ts/bazel_repository_test_dir' may only be called in bazel packages that correspond to the pnpm root package '' and pnpm workspace projects '' This was happening because the `.bazelrc` file only added `--deleted_packages` to the `build` command. We also need it for the `query` command. This second version of the patch fixes that. Original commit message: This patch migrates the current use of rules_nodejs to the new rules_js. rules_js is the intended replacement of rules_nodejs as per this note: https://github.com/aspect-build/rules_js#relationship-to-rules_nodejs > rules_js is an alternative to the build_bazel_rules_nodejs Bazel module > and accompanying npm packages hosted in > https://github.com/bazelbuild/rules_nodejs, which is now > unmaintained. All users are recommended to use rules_js instead. There are a few notable changes in this patch: 1. The `flatbuffer_ts_library` macro no longer accepts a `package_name` attribute. This is because rules_js appears to manage the import naming of dependencies via top-level `npm_link_package` targets. Users will have to migrate. 2. I added a few more arguments to `flatbuffer_library_public()`. These helped with exposing esbuild to `ts/compile_flat_file.sh`. 3. I pinned the version of `typescript` in `package.json` so that rules_ts can download the exact same version. rules_ts doesn't know what to do if the version isn't exact. 4. Since rules_js uses the pnpm locking mechanism, we now have a `pnpm-lock.yaml` file instead of a yarn lock file. 4. I added bazel targets for a few of the existing tests in `tests/ts`. They can be run with `bazel test //test/ts:all`. Since there is no flexbuffers bazel target, I did not add a bazel target for the corresponding test. 5. I added a separate workspace in `tests/ts/bazel_repository_test_dir/` to validate that the flatbuffers code can be imported as an external repository. You can run the test with `bazel test //test/ts:bazel_repository_test`. For this to work, I needed to expose a non-trivial chunk of the flatbuffers code to the test. I achieved this through some recursive `distribution` filegroups. This is inspired by rules_python's workspace tests. I did not do anything special to validate that the `gen_reflections` parameter works the same. This patch doesn't change anything about the TypeScript generation. As a side note: I am not an expert with rules_js. This patch is my attempt based on my limited understanding of the rule set. Fixes #7817 * Fix the query --------- Co-authored-by: Derek Bailey <derekbailey@google.com>
2023-05-03 18:48:15 +00:00
**kwargs
)
if reflection_name:
reflection_genrule_cmd = " ".join([
"SRCS=($(SRCS));",
"for f in $${SRCS[@]:0:%s}; do" % len(srcs),
[TS/JS] Entry point per namespace and reworked 1.x compatible single file build (#7510) * [TS/JS] Entry point per namespace * Fix handling of outputpath and array_test * Attempt to fix generate_code * Fix cwd for ts in generate_code * Attempt to fixup bazel and some docs * Add --ts-flat-files to bazel build to get bundle * Move to DEFAULT_FLATC_TS_ARGS * Attempt to add esbuild * Attempt to use npm instead * Remove futile attempt to add esbuild * Attempt to as bazel esbuild * Shuffle * Upgrade bazel deps * Revert failed attempts to get bazel working * Ignore flatc tests for now * Add esbuild dependency * `package.json` Include esbuild * `WORKSPACE` Add fetching esbuild binary * Update WORKSPACE * Unfreeze Lockfile * Update WORKSPACE * Update BUILD.bazel * Rework to suggest instead of running external bundler * Add esbuild generation to test script * Prelim bundle test * Run test JavaScriptTest from flatbuffers 1.x * Deps upgrade * Clang format fix * Revert bazel changes * Fix newline * Generate with type declarations * Handle "empty" root namespace * Adjust tests for typescript_keywords.ts * Separate test procedure for old node resolution module output * Fix rel path for root level re-exports * Bazel support for esbuild-based flatc Unfortunately, we lose typing information because the new esbuild method of generating single files does not generate type information. The method used here is a bit hack-ish because it relies on parsing the console output of flatc to figure out what to do. * Try to fix bazel build for when node isn't present on host * Auto formatting fixes * Fix missing generated code Co-authored-by: Derek Bailey <derekbailey@google.com> Co-authored-by: James Kuszmaul <jabukuszmaul+collab@gmail.com>
2023-01-21 20:22:22 +00:00
"$(location %s)" % (TRUE_FLATC_PATH),
"-b --schema",
" ".join(flatc_args),
" ".join(include_paths_cmd),
language_flag,
output_directory,
"$$f;",
"done",
])
reflection_outs = [
(out_prefix + "%s.bfbs") % (s.replace(".fbs", "").split("/")[-1])
for s in srcs
]
native.genrule(
name = "%s_srcs" % reflection_name,
srcs = srcs + includes,
outs = reflection_outs,
output_to_bindir = output_to_bindir,
[TS/JS] Entry point per namespace and reworked 1.x compatible single file build (#7510) * [TS/JS] Entry point per namespace * Fix handling of outputpath and array_test * Attempt to fix generate_code * Fix cwd for ts in generate_code * Attempt to fixup bazel and some docs * Add --ts-flat-files to bazel build to get bundle * Move to DEFAULT_FLATC_TS_ARGS * Attempt to add esbuild * Attempt to use npm instead * Remove futile attempt to add esbuild * Attempt to as bazel esbuild * Shuffle * Upgrade bazel deps * Revert failed attempts to get bazel working * Ignore flatc tests for now * Add esbuild dependency * `package.json` Include esbuild * `WORKSPACE` Add fetching esbuild binary * Update WORKSPACE * Unfreeze Lockfile * Update WORKSPACE * Update BUILD.bazel * Rework to suggest instead of running external bundler * Add esbuild generation to test script * Prelim bundle test * Run test JavaScriptTest from flatbuffers 1.x * Deps upgrade * Clang format fix * Revert bazel changes * Fix newline * Generate with type declarations * Handle "empty" root namespace * Adjust tests for typescript_keywords.ts * Separate test procedure for old node resolution module output * Fix rel path for root level re-exports * Bazel support for esbuild-based flatc Unfortunately, we lose typing information because the new esbuild method of generating single files does not generate type information. The method used here is a bit hack-ish because it relies on parsing the console output of flatc to figure out what to do. * Try to fix bazel build for when node isn't present on host * Auto formatting fixes * Fix missing generated code Co-authored-by: Derek Bailey <derekbailey@google.com> Co-authored-by: James Kuszmaul <jabukuszmaul+collab@gmail.com>
2023-01-21 20:22:22 +00:00
tools = [TRUE_FLATC_PATH],
compatible_with = compatible_with,
restricted_to = restricted_to,
target_compatible_with = target_compatible_with,
cmd = reflection_genrule_cmd,
message = "Generating flatbuffer reflection binary for %s:" % (name),
visibility = reflection_visibility,
)
native.filegroup(
name = "%s_out" % reflection_name,
srcs = reflection_outs,
2019-05-06 19:10:11 +00:00
visibility = reflection_visibility,
compatible_with = compatible_with,
restricted_to = restricted_to,
)
def flatbuffer_cc_library(
name,
srcs,
srcs_filegroup_name = "",
out_prefix = "",
deps = [],
includes = [],
[TS/JS] Entry point per namespace and reworked 1.x compatible single file build (#7510) * [TS/JS] Entry point per namespace * Fix handling of outputpath and array_test * Attempt to fix generate_code * Fix cwd for ts in generate_code * Attempt to fixup bazel and some docs * Add --ts-flat-files to bazel build to get bundle * Move to DEFAULT_FLATC_TS_ARGS * Attempt to add esbuild * Attempt to use npm instead * Remove futile attempt to add esbuild * Attempt to as bazel esbuild * Shuffle * Upgrade bazel deps * Revert failed attempts to get bazel working * Ignore flatc tests for now * Add esbuild dependency * `package.json` Include esbuild * `WORKSPACE` Add fetching esbuild binary * Update WORKSPACE * Unfreeze Lockfile * Update WORKSPACE * Update BUILD.bazel * Rework to suggest instead of running external bundler * Add esbuild generation to test script * Prelim bundle test * Run test JavaScriptTest from flatbuffers 1.x * Deps upgrade * Clang format fix * Revert bazel changes * Fix newline * Generate with type declarations * Handle "empty" root namespace * Adjust tests for typescript_keywords.ts * Separate test procedure for old node resolution module output * Fix rel path for root level re-exports * Bazel support for esbuild-based flatc Unfortunately, we lose typing information because the new esbuild method of generating single files does not generate type information. The method used here is a bit hack-ish because it relies on parsing the console output of flatc to figure out what to do. * Try to fix bazel build for when node isn't present on host * Auto formatting fixes * Fix missing generated code Co-authored-by: Derek Bailey <derekbailey@google.com> Co-authored-by: James Kuszmaul <jabukuszmaul+collab@gmail.com>
2023-01-21 20:22:22 +00:00
include_paths = None,
cc_include_paths = [],
flatc_args = DEFAULT_FLATC_ARGS,
visibility = None,
compatible_with = None,
restricted_to = None,
target_compatible_with = None,
srcs_filegroup_visibility = None,
gen_reflections = False):
"""A cc_library with the generated reader/writers for the given flatbuffer definitions.
Args:
name: Rule name.
srcs: Source .fbs files. Sent in order to the compiler.
srcs_filegroup_name: Name of the output filegroup that holds srcs. Pass this
filegroup into the `includes` parameter of any other
flatbuffer_cc_library that depends on this one's schemas.
out_prefix: Prepend this path to the front of all generated files. Usually
is a directory name.
deps: Optional, list of other flatbuffer_cc_library's to depend on. Cannot be specified
alongside includes.
includes: Optional, list of filegroups of schemas that the srcs depend on.
Use of this is discouraged, and may be deprecated.
include_paths: Optional, list of paths the includes files can be found in.
cc_include_paths: Optional, list of paths to add to the cc_library includes attribute.
flatc_args: Optional list of additional arguments to pass to flatc
(e.g. --gen-mutable).
visibility: The visibility of the generated cc_library. By default, use the
default visibility of the project.
srcs_filegroup_visibility: The visibility of the generated srcs filegroup.
By default, use the value of the visibility parameter above.
gen_reflections: Optional, if true this will generate the flatbuffer
reflection binaries for the schemas.
compatible_with: Optional, The list of environments this rule can be built
for, in addition to default-supported environments.
restricted_to: Optional, The list of environments this rule can be built
for, instead of default-supported environments.
target_compatible_with: Optional, The list of target platform constraints
to use.
This produces:
filegroup([name]_srcs): all generated .h files.
filegroup(srcs_filegroup_name if specified, or [name]_includes if not):
Other flatbuffer_cc_library's can pass this in for their `includes`
parameter, if they depend on the schemas in this library.
Fileset([name]_reflection): (Optional) all generated reflection binaries.
cc_library([name]): library with sources and flatbuffers deps.
"""
output_headers = [
(out_prefix + "%s_generated.h") % (s.replace(".fbs", "").split("/")[-1].split(":")[-1])
for s in srcs
]
if deps and includes:
# There is no inherent reason we couldn't support both, but this discourages
# use of includes without good reason.
fail("Cannot specify both deps and include in flatbuffer_cc_library.")
if deps:
includes = [d + "_includes" for d in deps]
reflection_name = "%s_reflection" % name if gen_reflections else ""
srcs_lib = "%s_srcs" % (name)
flatbuffer_library_public(
name = srcs_lib,
srcs = srcs,
outs = output_headers,
language_flag = "-c",
out_prefix = out_prefix,
includes = includes,
include_paths = include_paths,
flatc_args = flatc_args,
compatible_with = compatible_with,
restricted_to = restricted_to,
target_compatible_with = target_compatible_with,
reflection_name = reflection_name,
2019-05-06 19:10:11 +00:00
reflection_visibility = visibility,
)
cc_library(
name = name,
hdrs = [
":" + srcs_lib,
],
srcs = [
":" + srcs_lib,
],
features = [
"-parse_headers",
],
deps = [
"@com_github_google_flatbuffers//:runtime_cc",
"@com_github_google_flatbuffers//:flatbuffers",
] + deps,
includes = cc_include_paths,
compatible_with = compatible_with,
restricted_to = restricted_to,
target_compatible_with = target_compatible_with,
linkstatic = 1,
visibility = visibility,
)
# A filegroup for the `srcs`. That is, all the schema files for this
# Flatbuffer set.
native.filegroup(
name = srcs_filegroup_name if srcs_filegroup_name else "%s_includes" % (name),
srcs = srcs + includes,
compatible_with = compatible_with,
restricted_to = restricted_to,
visibility = srcs_filegroup_visibility if srcs_filegroup_visibility != None else visibility,
)