issues with IBM JVM on Linux

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

issues with IBM JVM on Linux

Alexey Loubyansky
I've been trying to narrow down sporadic failures in the wfcore patching
tests on the IBM JVM on Linux (version info below) for the last few
days. I wanted to share this because what I see doesn't make sense to
me. And I am not sure whether other modules have similar issues.

So, in patching tests the (sporadic) problem appears when parsing XML
attributes. E.g. if there is an element with attributes 'name' and
'add-on', sometimes the value of name will end up containing the value
of add-on attribute. Here is the actual code for this (Attribute is an
enum) from org.jboss.as.patching.metadata.PatchXmlUtils:

String name = null;
boolean isAddOn = false;
final int count = reader.getAttributeCount();
for (int i = 0; i < count; i++) {
     final String value = reader.getAttributeValue(i);
     final Attribute attribute =
Attribute.forName(reader.getAttributeLocalName(i));
     if(Attribute.NAME == attribute) {
         name = value;
     } else if (Attribute.ADD_ON == attribute) {
         isAddOn = Boolean.valueOf(value);
     } else {
         throw unexpectedAttribute(reader, i);
     }
}

So, sometimes the tests fail because the value of name is 'true',
apparently. This is just one example. It affects parsing attributes in
general in this class.

I noticed that if I do anything with variable 'value' in the loop (or
pass it as an argument to some method and actually do something with it
in that method), e.g. System.err.println("value=" + value); the problem
goes away.

Another workaround I've found is to inline the variable, i.e. use
reader.getAttributeValue(i) directly instead of using a variable.

By the "problem goes away" I mean that I haven't seen these kind of
failures anymore with the changes I mentioned. To give some statistics,
with the original code the tests would fail from 2 to 5 times out of 10
runs.
With the inlined variable value they pass 100%. I've be running them for
3 days now.

Here is the JVM info.

java version "1.8.0"
Java(TM) SE Runtime Environment (build pxi3280-20150129_02)
IBM J9 VM (build 2.8, JRE 1.8.0 Linux x86-32 20150116_231420 (JIT
enabled, AOT enabled)
J9VM - R28_Java8_GA_20150116_2030_B231420
JIT  - tr.r14.java_20150109_82886.02
GC   - R28_Java8_GA_20150116_2030_B231420
J9CL - 20150116_231420)
JCL - 20150123_01 based on Oracle jdk8u31-b12


Any ideas or suggestions?


Thanks,
Alexey
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: issues with IBM JVM on Linux

Scott Marlow
Are you using the latest JVM?  From [1], it looks like the last update
was on July 17th 2015.

Does it make a difference whether the classes were generated by the IBM
versus Oracle versus OpenJDK compiler?

Scott

[1] http://www.ibm.com/developerworks/java/jdk/linux/download.html

On 09/25/2015 09:08 AM, Alexey Loubyansky wrote:

> I've been trying to narrow down sporadic failures in the wfcore patching
> tests on the IBM JVM on Linux (version info below) for the last few
> days. I wanted to share this because what I see doesn't make sense to
> me. And I am not sure whether other modules have similar issues.
>
> So, in patching tests the (sporadic) problem appears when parsing XML
> attributes. E.g. if there is an element with attributes 'name' and
> 'add-on', sometimes the value of name will end up containing the value
> of add-on attribute. Here is the actual code for this (Attribute is an
> enum) from org.jboss.as.patching.metadata.PatchXmlUtils:
>
> String name = null;
> boolean isAddOn = false;
> final int count = reader.getAttributeCount();
> for (int i = 0; i < count; i++) {
>       final String value = reader.getAttributeValue(i);
>       final Attribute attribute =
> Attribute.forName(reader.getAttributeLocalName(i));
>       if(Attribute.NAME == attribute) {
>           name = value;
>       } else if (Attribute.ADD_ON == attribute) {
>           isAddOn = Boolean.valueOf(value);
>       } else {
>           throw unexpectedAttribute(reader, i);
>       }
> }
>
> So, sometimes the tests fail because the value of name is 'true',
> apparently. This is just one example. It affects parsing attributes in
> general in this class.
>
> I noticed that if I do anything with variable 'value' in the loop (or
> pass it as an argument to some method and actually do something with it
> in that method), e.g. System.err.println("value=" + value); the problem
> goes away.
>
> Another workaround I've found is to inline the variable, i.e. use
> reader.getAttributeValue(i) directly instead of using a variable.
>
> By the "problem goes away" I mean that I haven't seen these kind of
> failures anymore with the changes I mentioned. To give some statistics,
> with the original code the tests would fail from 2 to 5 times out of 10
> runs.
> With the inlined variable value they pass 100%. I've be running them for
> 3 days now.
>
> Here is the JVM info.
>
> java version "1.8.0"
> Java(TM) SE Runtime Environment (build pxi3280-20150129_02)
> IBM J9 VM (build 2.8, JRE 1.8.0 Linux x86-32 20150116_231420 (JIT
> enabled, AOT enabled)
> J9VM - R28_Java8_GA_20150116_2030_B231420
> JIT  - tr.r14.java_20150109_82886.02
> GC   - R28_Java8_GA_20150116_2030_B231420
> J9CL - 20150116_231420)
> JCL - 20150123_01 based on Oracle jdk8u31-b12
>
>
> Any ideas or suggestions?
>
>
> Thanks,
> Alexey
> _______________________________________________
> wildfly-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: issues with IBM JVM on Linux

jtgreene
Administrator
In reply to this post by Alexey Loubyansky
Sounds like bad opto. Can you try with -Xint?

> On Sep 25, 2015, at 8:09 AM, Alexey Loubyansky <[hidden email]> wrote:
>
> I've been trying to narrow down sporadic failures in the wfcore patching
> tests on the IBM JVM on Linux (version info below) for the last few
> days. I wanted to share this because what I see doesn't make sense to
> me. And I am not sure whether other modules have similar issues.
>
> So, in patching tests the (sporadic) problem appears when parsing XML
> attributes. E.g. if there is an element with attributes 'name' and
> 'add-on', sometimes the value of name will end up containing the value
> of add-on attribute. Here is the actual code for this (Attribute is an
> enum) from org.jboss.as.patching.metadata.PatchXmlUtils:
>
> String name = null;
> boolean isAddOn = false;
> final int count = reader.getAttributeCount();
> for (int i = 0; i < count; i++) {
>     final String value = reader.getAttributeValue(i);
>     final Attribute attribute =
> Attribute.forName(reader.getAttributeLocalName(i));
>     if(Attribute.NAME == attribute) {
>         name = value;
>     } else if (Attribute.ADD_ON == attribute) {
>         isAddOn = Boolean.valueOf(value);
>     } else {
>         throw unexpectedAttribute(reader, i);
>     }
> }
>
> So, sometimes the tests fail because the value of name is 'true',
> apparently. This is just one example. It affects parsing attributes in
> general in this class.
>
> I noticed that if I do anything with variable 'value' in the loop (or
> pass it as an argument to some method and actually do something with it
> in that method), e.g. System.err.println("value=" + value); the problem
> goes away.
>
> Another workaround I've found is to inline the variable, i.e. use
> reader.getAttributeValue(i) directly instead of using a variable.
>
> By the "problem goes away" I mean that I haven't seen these kind of
> failures anymore with the changes I mentioned. To give some statistics,
> with the original code the tests would fail from 2 to 5 times out of 10
> runs.
> With the inlined variable value they pass 100%. I've be running them for
> 3 days now.
>
> Here is the JVM info.
>
> java version "1.8.0"
> Java(TM) SE Runtime Environment (build pxi3280-20150129_02)
> IBM J9 VM (build 2.8, JRE 1.8.0 Linux x86-32 20150116_231420 (JIT
> enabled, AOT enabled)
> J9VM - R28_Java8_GA_20150116_2030_B231420
> JIT  - tr.r14.java_20150109_82886.02
> GC   - R28_Java8_GA_20150116_2030_B231420
> J9CL - 20150116_231420)
> JCL - 20150123_01 based on Oracle jdk8u31-b12
>
>
> Any ideas or suggestions?
>
>
> Thanks,
> Alexey
> _______________________________________________
> wildfly-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/wildfly-dev

_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: issues with IBM JVM on Linux

Alexey Loubyansky
In reply to this post by Scott Marlow
On 09/25/2015 03:18 PM, Scott Marlow wrote:
> Are you using the latest JVM?  From [1], it looks like the last update
> was on July 17th 2015.

No, I've been using the one we have installed in brontes to be able to
reproduce.

But I've tried the latest and it still fails occasionally although less
often: once or twice in 10 runs. So it appears to be better than the one
we've been using. Tomaz is going to upgrade it.

> Does it make a difference whether the classes were generated by the IBM
> versus Oracle versus OpenJDK compiler?

Haven't tried that yet.

Thanks,
Alexey

> Scott
>
> [1] http://www.ibm.com/developerworks/java/jdk/linux/download.html
>
> On 09/25/2015 09:08 AM, Alexey Loubyansky wrote:
>> I've been trying to narrow down sporadic failures in the wfcore patching
>> tests on the IBM JVM on Linux (version info below) for the last few
>> days. I wanted to share this because what I see doesn't make sense to
>> me. And I am not sure whether other modules have similar issues.
>>
>> So, in patching tests the (sporadic) problem appears when parsing XML
>> attributes. E.g. if there is an element with attributes 'name' and
>> 'add-on', sometimes the value of name will end up containing the value
>> of add-on attribute. Here is the actual code for this (Attribute is an
>> enum) from org.jboss.as.patching.metadata.PatchXmlUtils:
>>
>> String name = null;
>> boolean isAddOn = false;
>> final int count = reader.getAttributeCount();
>> for (int i = 0; i < count; i++) {
>>        final String value = reader.getAttributeValue(i);
>>        final Attribute attribute =
>> Attribute.forName(reader.getAttributeLocalName(i));
>>        if(Attribute.NAME == attribute) {
>>            name = value;
>>        } else if (Attribute.ADD_ON == attribute) {
>>            isAddOn = Boolean.valueOf(value);
>>        } else {
>>            throw unexpectedAttribute(reader, i);
>>        }
>> }
>>
>> So, sometimes the tests fail because the value of name is 'true',
>> apparently. This is just one example. It affects parsing attributes in
>> general in this class.
>>
>> I noticed that if I do anything with variable 'value' in the loop (or
>> pass it as an argument to some method and actually do something with it
>> in that method), e.g. System.err.println("value=" + value); the problem
>> goes away.
>>
>> Another workaround I've found is to inline the variable, i.e. use
>> reader.getAttributeValue(i) directly instead of using a variable.
>>
>> By the "problem goes away" I mean that I haven't seen these kind of
>> failures anymore with the changes I mentioned. To give some statistics,
>> with the original code the tests would fail from 2 to 5 times out of 10
>> runs.
>> With the inlined variable value they pass 100%. I've be running them for
>> 3 days now.
>>
>> Here is the JVM info.
>>
>> java version "1.8.0"
>> Java(TM) SE Runtime Environment (build pxi3280-20150129_02)
>> IBM J9 VM (build 2.8, JRE 1.8.0 Linux x86-32 20150116_231420 (JIT
>> enabled, AOT enabled)
>> J9VM - R28_Java8_GA_20150116_2030_B231420
>> JIT  - tr.r14.java_20150109_82886.02
>> GC   - R28_Java8_GA_20150116_2030_B231420
>> J9CL - 20150116_231420)
>> JCL - 20150123_01 based on Oracle jdk8u31-b12
>>
>>
>> Any ideas or suggestions?
>>
>>
>> Thanks,
>> Alexey
>> _______________________________________________
>> wildfly-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>
> _______________________________________________
> wildfly-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: issues with IBM JVM on Linux

Radoslaw Rodak
Hi

Few months ago I had pretty similar issue on IBM JDK , but version 1.7.0 .

Doing 10 time same xslt transformation randomly it was failing.

Problem was JIT Compiler Bug in IBM JDK…. issue was fix in newer Version of IBM JDK.
Best ist to get newest IBM JDK Version.

Work Arround for me on IBM JDK was to use  TransformerFactory , which is used as default on Oracle JDK on Linux and Windows, which is not the case on IBM JDK...:

-Djavax.xml.transform.TransformerFactory=org.apache.xalan.processor.TransformerFactoryImpl

To see if the error is really because of IBM JIT ( begin of this year IBM JDK had a lot of JIT bugfixes… ) you can also just turn off JIT ( -Xnojit ) and see if the problem still happened ( side effect test will run very slow )

P.S:

Interesting side Effect was, transformation was 50% faster with Xalan Implementation compared to default IBM TransformerFactory …

Cheers
Radek

Am 25.09.2015 um 18:39 schrieb Alexey Loubyansky <[hidden email]>:

> On 09/25/2015 03:18 PM, Scott Marlow wrote:
>> Are you using the latest JVM?  From [1], it looks like the last update
>> was on July 17th 2015.
>
> No, I've been using the one we have installed in brontes to be able to
> reproduce.
>
> But I've tried the latest and it still fails occasionally although less
> often: once or twice in 10 runs. So it appears to be better than the one
> we've been using. Tomaz is going to upgrade it.
>
>> Does it make a difference whether the classes were generated by the IBM
>> versus Oracle versus OpenJDK compiler?
>
> Haven't tried that yet.
>
> Thanks,
> Alexey
>
>> Scott
>>
>> [1] http://www.ibm.com/developerworks/java/jdk/linux/download.html
>>
>> On 09/25/2015 09:08 AM, Alexey Loubyansky wrote:
>>> I've been trying to narrow down sporadic failures in the wfcore patching
>>> tests on the IBM JVM on Linux (version info below) for the last few
>>> days. I wanted to share this because what I see doesn't make sense to
>>> me. And I am not sure whether other modules have similar issues.
>>>
>>> So, in patching tests the (sporadic) problem appears when parsing XML
>>> attributes. E.g. if there is an element with attributes 'name' and
>>> 'add-on', sometimes the value of name will end up containing the value
>>> of add-on attribute. Here is the actual code for this (Attribute is an
>>> enum) from org.jboss.as.patching.metadata.PatchXmlUtils:
>>>
>>> String name = null;
>>> boolean isAddOn = false;
>>> final int count = reader.getAttributeCount();
>>> for (int i = 0; i < count; i++) {
>>>       final String value = reader.getAttributeValue(i);
>>>       final Attribute attribute =
>>> Attribute.forName(reader.getAttributeLocalName(i));
>>>       if(Attribute.NAME == attribute) {
>>>           name = value;
>>>       } else if (Attribute.ADD_ON == attribute) {
>>>           isAddOn = Boolean.valueOf(value);
>>>       } else {
>>>           throw unexpectedAttribute(reader, i);
>>>       }
>>> }
>>>
>>> So, sometimes the tests fail because the value of name is 'true',
>>> apparently. This is just one example. It affects parsing attributes in
>>> general in this class.
>>>
>>> I noticed that if I do anything with variable 'value' in the loop (or
>>> pass it as an argument to some method and actually do something with it
>>> in that method), e.g. System.err.println("value=" + value); the problem
>>> goes away.
>>>
>>> Another workaround I've found is to inline the variable, i.e. use
>>> reader.getAttributeValue(i) directly instead of using a variable.
>>>
>>> By the "problem goes away" I mean that I haven't seen these kind of
>>> failures anymore with the changes I mentioned. To give some statistics,
>>> with the original code the tests would fail from 2 to 5 times out of 10
>>> runs.
>>> With the inlined variable value they pass 100%. I've be running them for
>>> 3 days now.
>>>
>>> Here is the JVM info.
>>>
>>> java version "1.8.0"
>>> Java(TM) SE Runtime Environment (build pxi3280-20150129_02)
>>> IBM J9 VM (build 2.8, JRE 1.8.0 Linux x86-32 20150116_231420 (JIT
>>> enabled, AOT enabled)
>>> J9VM - R28_Java8_GA_20150116_2030_B231420
>>> JIT  - tr.r14.java_20150109_82886.02
>>> GC   - R28_Java8_GA_20150116_2030_B231420
>>> J9CL - 20150116_231420)
>>> JCL - 20150123_01 based on Oracle jdk8u31-b12
>>>
>>>
>>> Any ideas or suggestions?
>>>
>>>
>>> Thanks,
>>> Alexey
>>> _______________________________________________
>>> wildfly-dev mailing list
>>> [hidden email]
>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>>
>> _______________________________________________
>> wildfly-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>
> _______________________________________________
> wildfly-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/wildfly-dev


_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev
Reply | Threaded
Open this post in threaded view
|

Re: issues with IBM JVM on Linux

Alexey Loubyansky
Thanks. I tried -Xnojit (the tests did take significantly more time to
complete) but it didn't affect the results. The testsuite failed 3 times
out of 5 runs with the same errors.

Alexey

On 09/25/2015 08:14 PM, Radoslaw Rodak wrote:

> Hi
>
> Few months ago I had pretty similar issue on IBM JDK , but version 1.7.0 .
>
> Doing 10 time same xslt transformation randomly it was failing.
>
> Problem was JIT Compiler Bug in IBM JDK…. issue was fix in newer Version of IBM JDK.
> Best ist to get newest IBM JDK Version.
>
> Work Arround for me on IBM JDK was to use  TransformerFactory , which is used as default on Oracle JDK on Linux and Windows, which is not the case on IBM JDK...:
>
> -Djavax.xml.transform.TransformerFactory=org.apache.xalan.processor.TransformerFactoryImpl
>
> To see if the error is really because of IBM JIT ( begin of this year IBM JDK had a lot of JIT bugfixes… ) you can also just turn off JIT ( -Xnojit ) and see if the problem still happened ( side effect test will run very slow )
>
> P.S:
>
> Interesting side Effect was, transformation was 50% faster with Xalan Implementation compared to default IBM TransformerFactory …
>
> Cheers
> Radek
>
> Am 25.09.2015 um 18:39 schrieb Alexey Loubyansky <[hidden email]>:
>
>> On 09/25/2015 03:18 PM, Scott Marlow wrote:
>>> Are you using the latest JVM?  From [1], it looks like the last update
>>> was on July 17th 2015.
>>
>> No, I've been using the one we have installed in brontes to be able to
>> reproduce.
>>
>> But I've tried the latest and it still fails occasionally although less
>> often: once or twice in 10 runs. So it appears to be better than the one
>> we've been using. Tomaz is going to upgrade it.
>>
>>> Does it make a difference whether the classes were generated by the IBM
>>> versus Oracle versus OpenJDK compiler?
>>
>> Haven't tried that yet.
>>
>> Thanks,
>> Alexey
>>
>>> Scott
>>>
>>> [1] http://www.ibm.com/developerworks/java/jdk/linux/download.html
>>>
>>> On 09/25/2015 09:08 AM, Alexey Loubyansky wrote:
>>>> I've been trying to narrow down sporadic failures in the wfcore patching
>>>> tests on the IBM JVM on Linux (version info below) for the last few
>>>> days. I wanted to share this because what I see doesn't make sense to
>>>> me. And I am not sure whether other modules have similar issues.
>>>>
>>>> So, in patching tests the (sporadic) problem appears when parsing XML
>>>> attributes. E.g. if there is an element with attributes 'name' and
>>>> 'add-on', sometimes the value of name will end up containing the value
>>>> of add-on attribute. Here is the actual code for this (Attribute is an
>>>> enum) from org.jboss.as.patching.metadata.PatchXmlUtils:
>>>>
>>>> String name = null;
>>>> boolean isAddOn = false;
>>>> final int count = reader.getAttributeCount();
>>>> for (int i = 0; i < count; i++) {
>>>>        final String value = reader.getAttributeValue(i);
>>>>        final Attribute attribute =
>>>> Attribute.forName(reader.getAttributeLocalName(i));
>>>>        if(Attribute.NAME == attribute) {
>>>>            name = value;
>>>>        } else if (Attribute.ADD_ON == attribute) {
>>>>            isAddOn = Boolean.valueOf(value);
>>>>        } else {
>>>>            throw unexpectedAttribute(reader, i);
>>>>        }
>>>> }
>>>>
>>>> So, sometimes the tests fail because the value of name is 'true',
>>>> apparently. This is just one example. It affects parsing attributes in
>>>> general in this class.
>>>>
>>>> I noticed that if I do anything with variable 'value' in the loop (or
>>>> pass it as an argument to some method and actually do something with it
>>>> in that method), e.g. System.err.println("value=" + value); the problem
>>>> goes away.
>>>>
>>>> Another workaround I've found is to inline the variable, i.e. use
>>>> reader.getAttributeValue(i) directly instead of using a variable.
>>>>
>>>> By the "problem goes away" I mean that I haven't seen these kind of
>>>> failures anymore with the changes I mentioned. To give some statistics,
>>>> with the original code the tests would fail from 2 to 5 times out of 10
>>>> runs.
>>>> With the inlined variable value they pass 100%. I've be running them for
>>>> 3 days now.
>>>>
>>>> Here is the JVM info.
>>>>
>>>> java version "1.8.0"
>>>> Java(TM) SE Runtime Environment (build pxi3280-20150129_02)
>>>> IBM J9 VM (build 2.8, JRE 1.8.0 Linux x86-32 20150116_231420 (JIT
>>>> enabled, AOT enabled)
>>>> J9VM - R28_Java8_GA_20150116_2030_B231420
>>>> JIT  - tr.r14.java_20150109_82886.02
>>>> GC   - R28_Java8_GA_20150116_2030_B231420
>>>> J9CL - 20150116_231420)
>>>> JCL - 20150123_01 based on Oracle jdk8u31-b12
>>>>
>>>>
>>>> Any ideas or suggestions?
>>>>
>>>>
>>>> Thanks,
>>>> Alexey
>>>> _______________________________________________
>>>> wildfly-dev mailing list
>>>> [hidden email]
>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>>>
>>> _______________________________________________
>>> wildfly-dev mailing list
>>> [hidden email]
>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>>
>> _______________________________________________
>> wildfly-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>
_______________________________________________
wildfly-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/wildfly-dev