Shouldn't InputStream/Files::readAllBytes throw something other than OutOfMemoryError?

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Shouldn't InputStream/Files::readAllBytes throw something other than OutOfMemoryError?

Anthony Vanelverdinghe
Files::readAllBytes is specified to throw an OutOfMemoryError "if an
array of the required size cannot be allocated, for example the file is
larger that 2G". Now in Java 9, InputStream::readAllBytes does the same.

However, this overloads the meaning of OutOfMemoryError: either "the JVM
is out of memory" or "the resultant array would require long-based indices".

In my opinion, this overloading is problematic, because:
- OutOfMemoryError has very clear semantics, and I don't see the link
between OOME and the fact that a resultant byte[] would need to be >2G.
If I have 5G of free heap space, and try to read a 3G file, I'd expect
something like an UnsupportedOperationException, but definitely not an
OutOfMemoryError.
- the former meaning is an actual Error, whereas the latter is an
Exception from which the application can recover.
- developers might be tempted to catch the OOME and retry to read the
file/input stream in chunks, no matter the cause of the OOME.

What was the rationale for using OutOfMemory here? And would it still be
possible to change this before Rampdown Phase 2?

Kind regards,
Anthony


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Shouldn't InputStream/Files::readAllBytes throw something other than OutOfMemoryError?

Noctarius
Hey Anthony,

The meaning is already overloaded, as "Cannot create native thread"
is also an OutOfMemoryError and in like 99% of the cases means
"Linux ran out of filehandles". The chance the OS really couldn't
allocate a thread for the reason of no main memory available is very
narrow :)

Chris

Am 3/12/2017 um 3:24 PM schrieb Anthony Vanelverdinghe:

> Files::readAllBytes is specified to throw an OutOfMemoryError "if
> an array of the required size cannot be allocated, for example the
> file is larger that 2G". Now in Java 9, InputStream::readAllBytes
> does the same.
>
> However, this overloads the meaning of OutOfMemoryError: either
> "the JVM is out of memory" or "the resultant array would require
> long-based indices".
>
> In my opinion, this overloading is problematic, because:
> - OutOfMemoryError has very clear semantics, and I don't see the
> link between OOME and the fact that a resultant byte[] would need
> to be >2G. If I have 5G of free heap space, and try to read a 3G
> file, I'd expect something like an UnsupportedOperationException,
> but definitely not an OutOfMemoryError.
> - the former meaning is an actual Error, whereas the latter is an
> Exception from which the application can recover.
> - developers might be tempted to catch the OOME and retry to read
> the file/input stream in chunks, no matter the cause of the OOME.
>
> What was the rationale for using OutOfMemory here? And would it
> still be possible to change this before Rampdown Phase 2?
>
> Kind regards,
> Anthony
>
>

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Shouldn't InputStream/Files::readAllBytes throw something other than OutOfMemoryError?

Anthony Vanelverdinghe
Hi Chris

Point well taken, but being unable to create a native thread is
definitely a VirtualMachineError, and personally I don't care whether
the JVM throws an OOME or any other kind of VirtualMachineError in that
case.

My point was that I don't see how being unable to return a result
because of a language limitation (i.e. arrays being indexed by integers,
and thus limited to 2G for byte[]), has anything to do with
OutOfMemoryError. I believe it would be much more logical to throw a
recoverable RuntimeException in this case (e.g.
java.lang.ArrayOverflowException, as an analog of
java.nio.BufferOverflowException).

Anthony

On 12/03/2017 15:53, Christoph Engelbert wrote:

> Hey Anthony,
>
> The meaning is already overloaded, as "Cannot create native thread"
> is also an OutOfMemoryError and in like 99% of the cases means
> "Linux ran out of filehandles". The chance the OS really couldn't
> allocate a thread for the reason of no main memory available is very
> narrow :)
>
> Chris
>
> Am 3/12/2017 um 3:24 PM schrieb Anthony Vanelverdinghe:
>> Files::readAllBytes is specified to throw an OutOfMemoryError "if
>> an array of the required size cannot be allocated, for example the
>> file is larger that 2G". Now in Java 9, InputStream::readAllBytes
>> does the same.
>>
>> However, this overloads the meaning of OutOfMemoryError: either
>> "the JVM is out of memory" or "the resultant array would require
>> long-based indices".
>>
>> In my opinion, this overloading is problematic, because:
>> - OutOfMemoryError has very clear semantics, and I don't see the
>> link between OOME and the fact that a resultant byte[] would need
>> to be >2G. If I have 5G of free heap space, and try to read a 3G
>> file, I'd expect something like an UnsupportedOperationException,
>> but definitely not an OutOfMemoryError.
>> - the former meaning is an actual Error, whereas the latter is an
>> Exception from which the application can recover.
>> - developers might be tempted to catch the OOME and retry to read
>> the file/input stream in chunks, no matter the cause of the OOME.
>>
>> What was the rationale for using OutOfMemory here? And would it
>> still be possible to change this before Rampdown Phase 2?
>>
>> Kind regards,
>> Anthony
>>
>>

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Shouldn't InputStream/Files::readAllBytes throw something other than OutOfMemoryError?

Noctarius
Hey Anthony,

Fair point though. Guess it might be just for consistency, with existing behavior. Anyhow your explanation makes sense, let’s hope it’ll be a temporary limitation as we’re seeing Arrays 2.0 on the horizon and most most most probably will offer 64bit array indexes :-)

Chris

> On 12. Mar 2017, at 18:31, Anthony Vanelverdinghe <[hidden email]> wrote:
>
> Hi Chris
>
> Point well taken, but being unable to create a native thread is definitely a VirtualMachineError, and personally I don't care whether the JVM throws an OOME or any other kind of VirtualMachineError in that case.
>
> My point was that I don't see how being unable to return a result because of a language limitation (i.e. arrays being indexed by integers, and thus limited to 2G for byte[]), has anything to do with OutOfMemoryError. I believe it would be much more logical to throw a recoverable RuntimeException in this case (e.g. java.lang.ArrayOverflowException, as an analog of java.nio.BufferOverflowException).
>
> Anthony
>
> On 12/03/2017 15:53, Christoph Engelbert wrote:
>> Hey Anthony,
>>
>> The meaning is already overloaded, as "Cannot create native thread"
>> is also an OutOfMemoryError and in like 99% of the cases means
>> "Linux ran out of filehandles". The chance the OS really couldn't
>> allocate a thread for the reason of no main memory available is very
>> narrow :)
>>
>> Chris
>>
>> Am 3/12/2017 um 3:24 PM schrieb Anthony Vanelverdinghe:
>>> Files::readAllBytes is specified to throw an OutOfMemoryError "if
>>> an array of the required size cannot be allocated, for example the
>>> file is larger that 2G". Now in Java 9, InputStream::readAllBytes
>>> does the same.
>>>
>>> However, this overloads the meaning of OutOfMemoryError: either
>>> "the JVM is out of memory" or "the resultant array would require
>>> long-based indices".
>>>
>>> In my opinion, this overloading is problematic, because:
>>> - OutOfMemoryError has very clear semantics, and I don't see the
>>> link between OOME and the fact that a resultant byte[] would need
>>> to be >2G. If I have 5G of free heap space, and try to read a 3G
>>> file, I'd expect something like an UnsupportedOperationException,
>>> but definitely not an OutOfMemoryError.
>>> - the former meaning is an actual Error, whereas the latter is an
>>> Exception from which the application can recover.
>>> - developers might be tempted to catch the OOME and retry to read
>>> the file/input stream in chunks, no matter the cause of the OOME.
>>>
>>> What was the rationale for using OutOfMemory here? And would it
>>> still be possible to change this before Rampdown Phase 2?
>>>
>>> Kind regards,
>>> Anthony
>>>
>>>
>

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Shouldn't InputStream/Files::readAllBytes throw something other than OutOfMemoryError?

Chris Hegarty
In reply to this post by Anthony Vanelverdinghe
Anthony,

Many of the Collection types throw OOME if requested to grow
greater than ~2GB. Likewise some operations of String and
StringBuilder. Though this behavior is not strictly part of
the  current specification, I suspect that it is the defacto
standard ( since the implementation has always behaved this
way ).

The java.lang.module.ModuleReader::read method is another
method that specifies the behavior if the returned type is
not capable of supporting very large amounts of data.

I agree that the use of OOME here is somewhat overloaded, but
it appears that we already well down this path, best to make
it clear and consistent in the spec.

-Chris.

On 12/03/17 14:24, Anthony Vanelverdinghe wrote:

> Files::readAllBytes is specified to throw an OutOfMemoryError "if an
> array of the required size cannot be allocated, for example the file is
> larger that 2G". Now in Java 9, InputStream::readAllBytes does the same.
>
> However, this overloads the meaning of OutOfMemoryError: either "the JVM
> is out of memory" or "the resultant array would require long-based
> indices".
>
> In my opinion, this overloading is problematic, because:
> - OutOfMemoryError has very clear semantics, and I don't see the link
> between OOME and the fact that a resultant byte[] would need to be >2G.
> If I have 5G of free heap space, and try to read a 3G file, I'd expect
> something like an UnsupportedOperationException, but definitely not an
> OutOfMemoryError.
> - the former meaning is an actual Error, whereas the latter is an
> Exception from which the application can recover.
> - developers might be tempted to catch the OOME and retry to read the
> file/input stream in chunks, no matter the cause of the OOME.
>
> What was the rationale for using OutOfMemory here? And would it still be
> possible to change this before Rampdown Phase 2?
>
> Kind regards,
> Anthony
>
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Shouldn't InputStream/Files::readAllBytes throw something other than OutOfMemoryError?

Peter Levart


On 03/13/2017 12:33 PM, Chris Hegarty wrote:

> Anthony,
>
> Many of the Collection types throw OOME if requested to grow
> greater than ~2GB. Likewise some operations of String and
> StringBuilder. Though this behavior is not strictly part of
> the  current specification, I suspect that it is the defacto
> standard ( since the implementation has always behaved this
> way ).
>
> The java.lang.module.ModuleReader::read method is another
> method that specifies the behavior if the returned type is
> not capable of supporting very large amounts of data.
>
> I agree that the use of OOME here is somewhat overloaded, but
> it appears that we already well down this path, best to make
> it clear and consistent in the spec.

What about (talking about JDK10 of course) creating OutOfMemoryError
subclasses to cover cases that don't pertain to Java heap memory?

Regards, Peter

>
> -Chris.
>
> On 12/03/17 14:24, Anthony Vanelverdinghe wrote:
>> Files::readAllBytes is specified to throw an OutOfMemoryError "if an
>> array of the required size cannot be allocated, for example the file is
>> larger that 2G". Now in Java 9, InputStream::readAllBytes does the same.
>>
>> However, this overloads the meaning of OutOfMemoryError: either "the JVM
>> is out of memory" or "the resultant array would require long-based
>> indices".
>>
>> In my opinion, this overloading is problematic, because:
>> - OutOfMemoryError has very clear semantics, and I don't see the link
>> between OOME and the fact that a resultant byte[] would need to be >2G.
>> If I have 5G of free heap space, and try to read a 3G file, I'd expect
>> something like an UnsupportedOperationException, but definitely not an
>> OutOfMemoryError.
>> - the former meaning is an actual Error, whereas the latter is an
>> Exception from which the application can recover.
>> - developers might be tempted to catch the OOME and retry to read the
>> file/input stream in chunks, no matter the cause of the OOME.
>>
>> What was the rationale for using OutOfMemory here? And would it still be
>> possible to change this before Rampdown Phase 2?
>>
>> Kind regards,
>> Anthony
>>
>>

Loading...