The JBIG2 format allows for ridiculously large images (4G x 4G
pixels!), which means that jbig2dec may use an enormous amount of
memory when it tries to decode them. OSS-fuzz currently restricts test
cases to 2.5Gbyte of memory per run, so the jbig2dec fuzzer
implemented a custom allocator to limit jbig2dec's memory usage to
1Gbyte. When the custom allocator runs out of memory jbig2dec
indicates an error, frees its resources and returns.
Limiting memory usage to 1Gbyte, below OSS-fuzz's limit of 2.5Gbyte,
eliminated the entire class of false positive OSS-fuzz issues
concerning out of memory situations. These were false positives in the
sense that a program using jbig2dec is in control of how much memory
jbig2dec uses, but the program must implement a custom jbig2dec
allocator and limit it to the desired amount of memory.
Another class of false positive OSS-fuzz issues remain; issues where
the image data still takes more than 25 seconds to process, causing an
OSS-fuzz timeout. These cases use less than 1Gbyte of memory, but
processing that amount of data may still take a long time. Since
processing time and data size are related, a program may limit the
amount of memory allotted to jbigdec's custom allocator to something
less than 1Gbyte to reduce processing time.
Running through a set of real world JBIG2 images shows that no more
than 20MByte is used to decode any of them and none take more then 25
seconds to decode on a desktop machine. To eliminate the class of
false positive OSS-fuzz timeout issues the fuzzer will now limit the
amount of memory to 32Mbyte with the hope that their processing time
will be reduced below 25 seconds.
Of course OSS-fuzz may still detect issues where jbig2dec gets caught
in an endless loop (or the processing time is long for a reason other
than data size). These are the issues we want OSS-fuzz to identify and
get fixed, since the parameters causing those timeouts are not in
control by a program using jbig2dec.
Previously the allocator handled small pointer values incorrectly
and prevented multiple executions (unexpectedly used when OSS-fuzz
uses LSAN to look for memory leaks). More over there were cases
where too large allocations were not caught and the allocator
needlessly kept track of peak allocation statistics.
To make that possible these things were also done:
* Rename variables for clarity.
* Rename functions to separate them from jbig2dec's own allocator API.
* Consistently use size_t for all memory statistics.
* Add helper function for checking peak memory use.
* Add helper function for when limit is reached.
* Add helper function for when allocations fail.
* Add helper function for memory statis at end of run (including leak check).
This will help with fixing jbig2dec issue 21675 reported by OSS-fuzz.
Apparently the maximum memory usage is 2.5Gbyte so a limit of 3Gbyte
is to liberal. Set limit to 1Gbyte to make sure that any extra memory
used by the fuzzer is allowed. 1Gbyte ought to be enough for most real
world images decoded by jbig2dec.
* [presubmit] Enforce language attribute in projectt.yaml to be always set.
* Update documentation, better presubmit check, new project template.
* add docstring to templates.py
* Add example values in the project.yaml template and remove python value for now
* Add "project: c++" to 256 projects
* format
* Add labels and selective_unpack sections to the presubmit check
* fix incorrect auto_ccs format in three projects
* fix nss emails after rebase
* [jbig2dec] Fix issues in library glue logic.
This addresses the out of memory condition exhibited in
project-jbig2dec bug #17168.
* [jbig2dec] Print peak memory usage.
This would be helpful when we're approaching OOM conditions.
Sebastian Rasmussen