Commit Graph

10392 Commits

Author SHA1 Message Date
Dongge Liu a80fd730e4
Revert "target_experiment: don't use gcloud. (#11701)" (#11727)
This reverts commit 66bcb5af81 to help
investigate a recent slowness and errors in OSS-Fuzz-Gen.
Related: https://github.com/google/oss-fuzz-gen/pull/177.

Specially, here is [the
error](https://pantheon.corp.google.com/logs/query;cursorTimestamp=2024-03-25T06:45:55.502596773Z;duration=P7D;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22oss-fuzz%22%0Aresource.labels.location%3D%22us-central1-c%22%0Aresource.labels.cluster_name%3D%22llm-experiment%22%0Aresource.labels.namespace_name%3D%22default%22%0Alabels.k8s-pod%2Fbatch_kubernetes_io%2Fcontroller-uid%3D%226d6e5719-618b-4e3d-a161-065c116ccfc0%22%20severity%3E%3DDEFAULT%0A--Hide%20similar%20entries%0A-%2528textPayload%3D~%22WARNING:root:Missing%20result%20JSON%20of%20benchmark%20instance:%20%5B%5E%20%3D%5Ct%5Cn%5Cr%5Cf%5C%22%5C%2528%5C%2529%5C%5B%5C%5D%5C%7C'%5D%2B%20-%20%2528%2528%3F:%5Cd%5B,.%5D%3F%2529*%5Cd%2529%22%2529%0A--End%20of%20hide%20similar%20entries%0Atimestamp%3D%222024-03-25T06:45:55.502596773Z%22%0AinsertId%3D%22fw412us1tsys1vfh%22?project=oss-fuzz)
shown in recent experiments:
```json
{
  "textPayload": "Traceback (most recent call last):\t  File \"/tmp/tmpi8uy26ij/infra/build/functions/target_experiment.py\", line 296, in <module>\t    main()\t  File \"/tmp/tmpi8uy26ij/infra/build/functions/target_experiment.py\", line 289, in main\t    run_experiment(args.project, args.target, args.args, args.upload_output_log,\t  File \"/tmp/tmpi8uy26ij/infra/build/functions/target_experiment.py\", line 255, in run_experiment\t    build_lib.wait_for_build(build_id, credentials, 'oss-fuzz')\t  File \"/tmp/tmpi8uy26ij/infra/build/functions/build_lib.py\", line 627, in wait_for_build\t    id=build_id).execute()\t                 ^^^^^^^^^\t  File \"/tmp/tmpi8uy26ij/venv/lib/python3.11/site-packages/googleapiclient/_helpers.py\", line 134, in positional_wrapper\t    return wrapped(*args, **kwargs)\t           ^^^^^^^^^^^^^^^^^^^^^^^^\t  File \"/tmp/tmpi8uy26ij/venv/lib/python3.11/site-packages/googleapiclient/http.py\", line 892, in execute\t    resp, content = _retry_request(\t                    ^^^^^^^^^^^^^^^\t  File \"/tmp/tmpi8uy26ij/venv/lib/python3.11/site-packages/googleapiclient/http.py\", line 204, in _retry_request\t    raise exception\t  File \"/tmp/tmpi8uy26ij/venv/lib/python3.11/site-packages/googleapiclient/http.py\", line 177, in _retry_request\t    resp, content = http.request(uri, method, *args, **kwargs)\t                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\t  File \"/tmp/tmpi8uy26ij/venv/lib/python3.11/site-packages/google_auth_httplib2.py\", line 209, in request\t    self.credentials.before_request(self._request, method, uri, request_headers)\t  File \"/tmp/tmpi8uy26ij/venv/lib/python3.11/site-packages/google/auth/credentials.py\", line 133, in before_request\t    self.refresh(request)\t  File \"/tmp/tmpi8uy26ij/venv/lib/python3.11/site-packages/google/auth/compute_engine/credentials.py\", line 99, in refresh\t    self._retrieve_info(request)\t  File \"/tmp/tmpi8uy26ij/venv/lib/python3.11/site-packages/google/auth/compute_engine/credentials.py\", line 79, in _retrieve_info\t    info = _metadata.get_service_account_info(\t           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\t  File \"/tmp/tmpi8uy26ij/venv/lib/python3.11/site-packages/google/auth/compute_engine/_metadata.py\", line 227, in get_service_account_info\t    return get(\t           ^^^^\t  File \"/tmp/tmpi8uy26ij/venv/lib/python3.11/site-packages/google/auth/compute_engine/_metadata.py\", line 146, in get\t    response = request(url=url, method=\"GET\", headers=_METADATA_HEADERS)\t               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\t  File \"/tmp/tmpi8uy26ij/venv/lib/python3.11/site-packages/google_auth_httplib2.py\", line 119, in __call__\t    response, data = self.http.request(\t                     ^^^^^^^^^^^^^^^^^^\t  File \"/tmp/tmpi8uy26ij/venv/lib/python3.11/site-packages/httplib2/__init__.py\", line 1724, in request\t    (response, content) = self._request(\t                          ^^^^^^^^^^^^^^\t  File \"/tmp/tmpi8uy26ij/venv/lib/python3.11/site-packages/httplib2/__init__.py\", line 1444, in _request\t    (response, content) = self._conn_request(conn, request_uri, method, body, headers)\t                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\t  File \"/tmp/tmpi8uy26ij/venv/lib/python3.11/site-packages/httplib2/__init__.py\", line 1367, in _conn_request\t    conn.request(method, request_uri, body, headers)\t  File \"/usr/lib/python3.11/http/client.py\", line 1282, in request\t    self._send_request(method, url, body, headers, encode_chunked)\t  File \"/usr/lib/python3.11/http/client.py\", line 1328, in _send_request\t    self.endheaders(body, encode_chunked=encode_chunked)\t  File \"/usr/lib/python3.11/http/client.py\", line 1277, in endheaders\t    self._send_output(message_body, encode_chunked=encode_chunked)\t  File \"/usr/lib/python3.11/http/client.py\", line 1037, in _send_output\t    self.send(msg)\t  File \"/usr/lib/python3.11/http/client.py\", line 998, in send\t    self.sock.sendall(data)\tBrokenPipeError: [Errno 32] Broken pipe\t",
  "insertId": "fw412us1tsys1vfh",
  "resource": {
    "type": "k8s_container",
    "labels": {
      "location": "us-central1-c",
      "namespace_name": "default",
      "pod_name": "ofg-pr-177-dg-cxszc",
      "cluster_name": "llm-experiment",
      "container_name": "experiment",
      "project_id": "oss-fuzz"
    }
  },
  "timestamp": "2024-03-25T06:45:55.502596773Z",
  "severity": "ERROR",
  "labels": {
    "k8s-pod/batch_kubernetes_io/controller-uid": "6d6e5719-618b-4e3d-a161-065c116ccfc0",
    "k8s-pod/job-name": "ofg-pr-177-dg",
    "compute.googleapis.com/resource_name": "gke-llm-experiment-ofg-pr-exp-2621ef4f-sms5",
    "k8s-pod/batch_kubernetes_io/job-name": "ofg-pr-177-dg",
    "k8s-pod/controller-uid": "6d6e5719-618b-4e3d-a161-065c116ccfc0"
  },
  "logName": "projects/oss-fuzz/logs/stderr",
  "receiveTimestamp": "2024-03-25T06:45:57.681367653Z"
}
```
2024-03-25 23:39:11 +11:00
Oliver Chang d2d9728a9d
Handle BrokenPipeError in wait_for_build (#11729)
The GCE metadata server can be unresponsive sometimes, resulting in this
exception.

This should address the root cause of the stack trace pointed out in
https://github.com/google/oss-fuzz/pull/11727
2024-03-25 19:36:29 +11:00
Sergey Bronnikov 9613e6f8aa
lua: fix .dict and .options filenames (#11721)
OSS Fuzz documentation says that default name of dictionary file is the
same as fuzzer filename:

> If the dict filename is the same as your target binary name
> (i.e. `%fuzz_target%.dict`), it will be automatically used.

The patch fixes `build.sh` accordingly.
2024-03-23 20:51:36 +00:00
DavidKorczynski 3659b194b9
infra: bazel_build_fuzz_tests: do not force O0 (second try) (#11718)
This does the same as https://github.com/google/oss-fuzz/pull/11715
which seems to be stuck in the CI
2024-03-21 23:37:40 +00:00
Evgeny Vereshchagin 9c0495a280
Revert "[CFL] Temporary work around MSAN issue." (#11717)
This reverts commit d9a8b11259.

It should be safe to revert it now that
https://github.com/actions/runner-images/issues/9491 is closed.

That stopgap was also reverted in

ae0e1cb989
and the CI confirmed that other workflows with sanitizers are fine so
CIFuzz/CFLite should be fine as well.
2024-03-21 14:32:30 -04:00
RBRi 4a290f7bfd
fix pgp patch (#11719)
build failed because of the plugin update - please apply
hope this fixes the build again
2024-03-21 14:31:56 -04:00
Julien Voisin 5d1bf13462
Fix the xpdf fuzzer (#11716)
The number of pages is starting at `1`, and not at `0`

This should fix #11711
2024-03-21 10:59:22 +00:00
Oliver Siegmar cf33d039f9
fastcsv: Fix Coverage build failure (#11692)
This PR fixes a Coverage build failure reported in
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=67425.

The failure is due to missing Java 21 support in the coverage container
when running JaCoCo:

```
Caused by: java.lang.IllegalArgumentException: Unsupported class file major version 65
```

Unfortunately, I couldn't find a way to add Java 21 support. As a
workaround, I've downgraded the Java version to 17 when building the
project.

I've tested the changes locally and the Coverage build is now passing.

If you have an idea how to add Java 21 support instead of downgrading to
17, please let me know.
2024-03-21 10:58:45 +00:00
Dongge Liu 153df7202e
Unify the bucket directory name for target binary in `target_experiment.py` (#11713)
Related: https://github.com/google/oss-fuzz-gen/pull/172, #11700.
This simplifies the link generation in benchmark `JSON`.
Previously, the binary directory name matches the binary name, now it is
always called `target_binary/`.
2024-03-20 15:29:59 +11:00
jonathanmetzman 25ced0daa1
[CIFuzz] Make writing stacks in sarif more robust (#11710)
Also fix bug in test project.
2024-03-19 13:46:21 -04:00
Jonathan Metzman d9a8b11259 [CFL] Temporary work around MSAN issue.
MSAN recently broke on github actions. Add a command that works
around the issue to temporarily unbreak MSAN.
Remove this commit when the issue is fixed later this week.

Fixes: https://github.com/google/oss-fuzz/issues/11703
2024-03-19 09:39:27 -04:00
Jonathan Metzman 2e4d4398d9 Reapply "Fix MSAN on github actions (#11708)"
This reverts commit 33c4a6d4fa.
2024-03-19 09:35:05 -04:00
Dongge Liu 6bc246b980
`target_experiment.py` uploads reproducer and binary in target_experiment.py (#11700)
Helps https://github.com/google/oss-fuzz-gen/issues/156:

1. `target_experiment.py` takes a new parameter,
`upload_reproducer_path`.
2. `target_experiment.py` saves crash reproducer to
`local_artifect_path`.
3. `target_experiment.py` uploads the fuzz target binary and the crash
reproducer to bucket directory `upload_reproducer_path`.
2024-03-19 11:20:25 +00:00
Jonathan Metzman 33c4a6d4fa Revert "Fix MSAN on github actions (#11708)"
This reverts commit da734e0d3c.

Don't want the risk of breaking everything just to fix MSAN on github actions,
without more testing.
2024-03-18 22:45:54 -04:00
jonathanmetzman da734e0d3c
Fix MSAN on github actions (#11708) 2024-03-18 15:24:30 -04:00
Lu Xiaoyu 3bdb41cb42
[spdm-rs]: add initial integration for spdm-rs (#11671)
The `spdm-rs` provides a Rust language implementation of
[SPDM](https://www.dmtf.org/standards/spdm) which is used to for Trusted
Execution Environment I/O (TEE-I/O) in Confidential Computing.

Project repo: https://github.com/ccc-spdm-tools/spdm-rs

Signed-off-by: xiaoyuxlu <xiaoyu1.lu@intel.com>
2024-03-18 14:19:39 +00:00
Gilles Peskine 3eae524f2a
mbedtls now has a submodule (again) (#11694)
Fix the mbedtls build in bignum-fuzzer and cryptofuzz. (mbedtls and
ecc-diff-fuzzer still had `--recursive` from earlier days.)

Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
2024-03-18 10:13:25 -04:00
Daniel Salzman e54dcd3159
[knot-dns] fix compilation against GnuTLS (#11686)
Signed-off-by: Daniel Salzman <daniel.salzman@nic.cz>
2024-03-18 10:12:52 -04:00
Evgeny Vereshchagin 223a616f1b
[systemd] drop address confusing monorail (#11702)
monorail keeps updating every open issue every other day and also keeps
sending notifications to everyone.

To judge from https://github.com/google/oss-fuzz/issues/6825 it happens
due to inactive addresses and the only way to fix it is to remove
addresses like that.
2024-03-18 10:10:12 -04:00
Sebastian Rasmussen 4e87465e18
[mupdf] Adapt to new error reporting API and shorten build time. (#11696) 2024-03-18 12:01:03 +00:00
Szabolcs Horvát 2278aba7cd
igraph: provide access for an additional team member (#11699)
This is to give @GroteGnoom access to igraph crash reports, as they'll
be helping with fixing issues and adding new fuzz targets.
2024-03-18 12:00:34 +00:00
Li-Yu Yu 80eeefdf68
projects/cras: Remove unused deps (#11705)
ladspa-sdk support removed in https://crrev.com/c/5369016.
2024-03-18 12:00:18 +00:00
Zhiqiang Zhao 034d789c64
libspdm: update the project configuration. (#11704)
Signed-off-by: Zhao, Zhiqiang <zhiqiang.zhao@intel.com>
2024-03-18 12:00:04 +00:00
Oliver Chang 66bcb5af81
target_experiment: don't use gcloud. (#11701)
Instead, tar up the project directory ourselves, upload this to GCS, and
create the Build ourselves.

This is also a lot more efficient, as we're excluding a lot of
unnecessary dirs from being uploaded.

Fixes: https://github.com/google/oss-fuzz-gen/issues/163
2024-03-18 12:02:52 +11:00
Nick Wellnhofer 8bf8f269ad
libxml2: Move build.sh to upstream repo (#11691)
build.sh is now maintained in the upstream libxml2 repo to facilitate
changes.
2024-03-16 16:43:08 +00:00
Vedant Shrotria dc543015c4
[LITMUS]: Integrating more repositories under litmuschaos project (#11684)
Signed-off-by: Jonsy13 <vedant.shrotria@harness.io>
2024-03-15 13:11:57 -04:00
Nico Weber 9191c03efb
serenity: Add myself to the mailing list (#11683) 2024-03-15 13:11:20 -04:00
Oliver Siegmar 94f13c3674
fastcsv: Catch CsvParseException (#11685)
This PR fixes a false positive issues reported in
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=67386.
2024-03-14 08:40:30 -04:00
Daniel Novomeský 2ccd3b742e
kimageformats: upgrade libaom, libavif, libjxl (#11682) 2024-03-13 21:27:21 +00:00
DavidKorczynski 54cf7a92d1
infra: downgrade rust (#11681)
Downgrades Rust to nightly at 28th Dec 2023. The version that is build
is:

```sh
 > [2/2] RUN install_rust.sh:                                                                                                                                                                                                                                                          
0.210 + curl https://sh.rustup.rs                                                                                                                                                                                                                                                      
0.210 + sh -s -- -y --default-toolchain=nightly-2023-12-28 --profile=minimal                                                                                                                                                                                                           
0.217   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                                                                                                                                                                                  
0.217                                  Dload  Upload   Total   Spent    Left  Speed                                                                                                                                                                                                    
100 26495  100 26495    0     0   205k      0 --:--:-- --:--:-- --:--:--  206k                                                                                                                                                                                                         
0.348 info: downloading installer                                                                                                                                                                                                                                                      
1.603 warning: it looks like you have an existing installation of Rust at:                                                                                                                                                                                                             
1.603 warning: /usr/local/bin                                                                                                                                                                                                                                                          
1.603 warning: It is recommended that rustup be the primary Rust installation.                                                                                                                                                                                                         
1.603 warning: Otherwise you may have confusion unless you are careful with your PATH                                                                                                                                                                                                  
1.603 warning: If you are sure that you want both rustup and your already installed Rust                                                                                                                                                                                               
1.603 warning: then please reply `y' or `yes' or set RUSTUP_INIT_SKIP_PATH_CHECK to yes                                                                                                                                                                                                
1.603 warning: or pass `-y' to ignore all ignorable checks.                                                                                                                                                                                                                            
1.603 error: cannot install while Rust is installed                                                                                                                                                                                                                                    
1.603 warning: continuing (because the -y flag is set and the error is ignorable)                                                                                                                                                                                                      
1.622 info: profile set to 'minimal'                                                                                                                                                                                                                                                   
1.622 info: default host triple is x86_64-unknown-linux-gnu                                                                                                                                                                                                                            
1.622 info: syncing channel updates for 'nightly-2023-12-28-x86_64-unknown-linux-gnu'                                                                                                                                                                                                  
2.595 info: latest update on 2023-12-28, rust version 1.77.0-nightly (89e2160c4 2023-12-27)                                                                                                                                                                                            
2.595 info: downloading component 'cargo'                                                                                                                                                                                                                                              
3.274 info: downloading component 'rust-std'      
```

---------

Signed-off-by: David Korczynski <david@adalogics.com>
2024-03-13 13:12:26 +00:00
Oliver Siegmar f6c57d4cf9
FastCSV: initial integration (#11634)
Initial integration of [FastCSV](https://github.com/osiegmar/FastCSV).
2024-03-13 02:41:59 +00:00
Zhiqiang Zhao af35a02426
libspdm: Initial integration (#11672)
libspdm is a sample implementation that follows the DMTF [SPDM
](https://www.dmtf.org/standards/spdm)specifications.
The SPDM standard enables authentication, attestation and key exchange
to assist in providing infrastructure security enablement.

Signed-off-by: Zhao, Zhiqiang <zhiqiang.zhao@intel.com>
2024-03-12 15:16:38 +00:00
Michael Niedermayer fc56a2cfee
[FFmpeg] build.sh: Add tools/target_sws_fuzzer (#11629)
Connect a fuzzer thingy specifically for libswscale

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2024-03-12 11:14:50 -04:00
Robert Löhning 3143848f3f
[qt] Remove workaround which fixed build (#11679)
The issue is fixed upstream.
2024-03-12 11:09:29 -04:00
Vedant Shrotria 96900de249
[Litmus]: Added changes to run build.sh from source in litmuschaos project (#11680)
Signed-off-by: Jonsy13 <vedant.shrotria@harness.io>
2024-03-12 20:27:19 +11:00
DavidKorczynski 10e6cbd71d
infra: bump introspector (#11678)
Bumps introspector. The primary update is that we now expose all debug
information in an index file where each type is indexed by its address.
The address is the key used by LLVM, which is unique for each type.
There are additional helped features to make debug information look
prettier as it's quite verbose.

The goal is to assist making it easier to extract full type contexts.

Signed-off-by: David Korczynski <david@adalogics.com>
2024-03-10 10:55:06 +00:00
Fabian Meumertzheim c0c4b5402d
Fix rules_fuzzing presubmit (#11675)
Work around https://github.com/bazelbuild/bazel/issues/21592 as
discovered in
https://github.com/bazelbuild/rules_fuzzing/pull/242#issuecomment-1974913042.
2024-03-07 21:25:52 +00:00
Evgeny Vereshchagin 9d034750a9
[systemd] more contributors (#11676)
Add Ronan as discussed in
https://github.com/systemd/systemd/pull/30952#issuecomment-1982633296

Should finally address https://github.com/google/oss-fuzz/issues/10637
2024-03-07 18:33:06 +00:00
Liu Wei 1bdba6f59f
Fix Dockerfile and script to support td-shim & migtd & vtpm-td (#11673)
Signed-off-by: Wei Liu <wei3.liu@intel.com>
2024-03-06 15:19:12 +00:00
Tomas Tauber 2905836df2
pest: add rust-src to Dockerfile toolchain (#11674)
should fix https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=66887
2024-03-06 14:25:03 +00:00
Liu Wei 5f7304451b
MigTD: Initial integration (#11642)
Signed-off-by: Wei Liu <wei3.liu@intel.com>
2024-03-06 02:12:59 +00:00
pyoor 35f33bd726
servo: add rust-cssparser target (#11640)
Adds a fuzz target for the servo rust-cssparser.
2024-03-06 09:56:36 +11:00
James Zern 4af154a356
lib{aom,vpx}/project.yaml: update cc list (#11664)
add jingning@, jianj@ and wtc@ consistently; remove departed team member
johannkoenig@.
2024-03-06 09:55:35 +11:00
Harish Mahendrakar 779b066ff0
[libavc] Update auto_ccs (#11668)
Updated auto_ccs for libavc
2024-03-06 09:52:25 +11:00
Liu Wei c097401b4e
vtpm-td: Initial integration (#11647)
Signed-off-by: Wei Liu <wei3.liu@intel.com>
2024-03-06 09:49:20 +11:00
Liu Wei 030afb447c
td-shim: Initial integration (#11646)
TD Shim is a simplified [TDX virtual
firmware](https://github.com/confidential-containers/td-shim/blob/main/doc/tdshim_spec.md#vfw)
for the simplified kernel for TD container.

Signed-off-by: Wei Liu <wei3.liu@intel.com>
2024-03-06 09:48:47 +11:00
DavidKorczynski ab62cc0227
infra: bump fuzz introspector (#11667)
Contains a fix for a function-signature releated issue.

Signed-off-by: David Korczynski <david@adalogics.com>
2024-03-05 19:51:27 +00:00
Kanstantsin Sokal e30ab16ea5
[quickjs] Set max stack limit. (#11641)
Google's internal fuzzers found tests that make `fuzz_regexp.c` exceed
stack memory. Turns out, QuickJS already has built-in mechanism for
tracking that stack size doesn't exceed a given limit - it's just that
limit isn't set for fuzzer tests by default

Setting the limit to 64kb as a reasonable default setting
2024-03-05 19:43:39 +00:00
Moritz Firsching 8ff6d3901b
remove unused gflags dependency for libjxl (#11666)
revert #7063, since it is no longer needed, see
https://github.com/libjxl/libjxl/pull/1588.
2024-03-05 11:39:25 +00:00
Adnan Keskin cd700ebd69
Delete projects/markdown-it-js directory (#11612)
Because can't connect projects upstream, I will delete this project.
Thanks cluster fuzz team!
2024-03-05 11:12:44 +11:00