Skip to content

Flaky-test: org.apache.pulsar.broker.loadbalance.LoadBalancerTest.testLeaderElection #9408

Closed
#9443
Closed
Flaky-test: org.apache.pulsar.broker.loadbalance.LoadBalancerTest.testLeaderElection#9408
#9443
Assignees
merlimat
Labels
help wantedtype/bugThe PR fixed a bug or issue reported a bugtype/flaky-tests
Milestone
2.8.0

Description

@lhotari
lhotari
opened on Feb 1, 2021

org.apache.pulsar.broker.loadbalance.LoadBalancerTest is flaky. The testLeaderElection test method fails sporadically.

java.util.NoSuchElementException: No value present
	at java.util.Optional.get(Optional.java:135)
	at org.apache.pulsar.broker.loadbalance.LoadBalancerTest.loopUntilLeaderChanges(LoadBalancerTest.java:185)
	at org.apache.pulsar.broker.loadbalance.LoadBalancerTest.testLeaderElection(LoadBalancerTest.java:743)

Usage tip: To enable automatic navigation to failure message, open the links with CTRL/CMD-click.
example failure 2021-02-01T13:40:12.3522919Z
example failure 2021-02-01T11:54:58.8604105Z
example failure 2021-02-01T11:23:42.3794621Z
example failure 2021-02-01T11:06:52.7534377Z
example failure 2021-02-01T03:33:28.1828410Z
example failure 2021-01-31T13:45:59.5870974Z
example failure 2021-01-30T18:37:49.6338778Z
example failure 2021-01-30T18:29:54.3189511Z
example failure 2021-01-30T10:02:27.9882474Z
example failure 2021-01-30T09:44:03.3715367Z
example failure 2021-01-30T09:11:24.3449948Z
example failure 2021-01-30T08:53:01.0720954Z
example failure 2021-01-29T00:11:58.6667897Z
example failure 2021-01-28T15:32:42.7100802Z
example failure 2021-01-28T15:23:10.6157274Z
example failure 2021-01-28T15:12:14.1950625Z
example failure 2021-01-28T07:23:32.7249440Z
example failure 2021-01-28T07:15:33.8671272Z
example failure 2021-01-28T07:07:30.7523687Z
example failure 2021-01-28T01:59:46.6547905Z
example failure 2021-01-28T01:52:11.4000197Z
example failure 2021-01-27T21:58:30.2044042Z
example failure 2021-01-27T12:06:15.0002925Z
example failure 2021-01-27T11:58:41.8274891Z

Full exception stacktrace 
java.util.NoSuchElementException: No value present
	at java.util.Optional.get(Optional.java:135)
	at org.apache.pulsar.broker.loadbalance.LoadBalancerTest.loopUntilLeaderChanges(LoadBalancerTest.java:185)
	at org.apache.pulsar.broker.loadbalance.LoadBalancerTest.testLeaderElection(LoadBalancerTest.java:743)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
	at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:45)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:73)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


java.lang.AssertionError: expected [Optional[LeaderBroker(serviceUrl=http://localhost4:46725)]] but found [Optional.empty]
	at org.testng.Assert.fail(Assert.java:99)
	at org.testng.Assert.failNotEquals(Assert.java:1037)
	at org.testng.Assert.assertEqualsImpl(Assert.java:140)
	at org.testng.Assert.assertEquals(Assert.java:122)
	at org.testng.Assert.assertEquals(Assert.java:617)
	at org.apache.pulsar.broker.loadbalance.LoadBalancerTest.testLeaderElection(LoadBalancerTest.java:737)

Usage tip: To enable automatic navigation to failure message, open the links with CTRL/CMD-click.
example failure 2021-02-01T10:49:19.1817633Z

Full exception stacktrace 
java.lang.AssertionError: expected [Optional[LeaderBroker(serviceUrl=http://localhost4:46725)]] but found [Optional.empty]
	at org.testng.Assert.fail(Assert.java:99)
	at org.testng.Assert.failNotEquals(Assert.java:1037)
	at org.testng.Assert.assertEqualsImpl(Assert.java:140)
	at org.testng.Assert.assertEquals(Assert.java:122)
	at org.testng.Assert.assertEquals(Assert.java:617)
	at org.apache.pulsar.broker.loadbalance.LoadBalancerTest.testLeaderElection(LoadBalancerTest.java:737)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
	at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:45)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:73)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


java.lang.AssertionError: expected [Optional[LeaderBroker(serviceUrl=http://localhost4:35563)]] but found [Optional.empty]
	at org.testng.Assert.fail(Assert.java:99)
	at org.testng.Assert.failNotEquals(Assert.java:1037)
	at org.testng.Assert.assertEqualsImpl(Assert.java:140)
	at org.testng.Assert.assertEquals(Assert.java:122)
	at org.testng.Assert.assertEquals(Assert.java:617)
	at org.apache.pulsar.broker.loadbalance.LoadBalancerTest.testLeaderElection(LoadBalancerTest.java:737)

Usage tip: To enable automatic navigation to failure message, open the links with CTRL/CMD-click.
example failure 2021-02-01T13:27:59.9104169Z

Full exception stacktrace 
java.lang.AssertionError: expected [Optional[LeaderBroker(serviceUrl=http://localhost4:35563)]] but found [Optional.empty]
	at org.testng.Assert.fail(Assert.java:99)
	at org.testng.Assert.failNotEquals(Assert.java:1037)
	at org.testng.Assert.assertEqualsImpl(Assert.java:140)
	at org.testng.Assert.assertEquals(Assert.java:122)
	at org.testng.Assert.assertEquals(Assert.java:617)
	at org.apache.pulsar.broker.loadbalance.LoadBalancerTest.testLeaderElection(LoadBalancerTest.java:737)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
	at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:45)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:73)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


java.lang.AssertionError: expected [Optional[LeaderBroker(serviceUrl=http://localhost3:38237)]] but found [Optional.empty]
	at org.testng.Assert.fail(Assert.java:99)
	at org.testng.Assert.failNotEquals(Assert.java:1037)
	at org.testng.Assert.assertEqualsImpl(Assert.java:140)
	at org.testng.Assert.assertEquals(Assert.java:122)
	at org.testng.Assert.assertEquals(Assert.java:617)
	at org.apache.pulsar.broker.loadbalance.LoadBalancerTest.testLeaderElection(LoadBalancerTest.java:737)

Usage tip: To enable automatic navigation to failure message, open the links with CTRL/CMD-click.
example failure 2021-01-31T13:27:37.9313406Z

Full exception stacktrace 
java.lang.AssertionError: expected [Optional[LeaderBroker(serviceUrl=http://localhost3:38237)]] but found [Optional.empty]
	at org.testng.Assert.fail(Assert.java:99)
	at org.testng.Assert.failNotEquals(Assert.java:1037)
	at org.testng.Assert.assertEqualsImpl(Assert.java:140)
	at org.testng.Assert.assertEquals(Assert.java:122)
	at org.testng.Assert.assertEquals(Assert.java:617)
	at org.apache.pulsar.broker.loadbalance.LoadBalancerTest.testLeaderElection(LoadBalancerTest.java:737)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
	at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:45)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:73)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


java.lang.AssertionError: expected [Optional[LeaderBroker(serviceUrl=http://localhost4:36589)]] but found [Optional.empty]
	at org.testng.Assert.fail(Assert.java:99)
	at org.testng.Assert.failNotEquals(Assert.java:1037)
	at org.testng.Assert.assertEqualsImpl(Assert.java:140)
	at org.testng.Assert.assertEquals(Assert.java:122)
	at org.testng.Assert.assertEquals(Assert.java:617)
	at org.apache.pulsar.broker.loadbalance.LoadBalancerTest.testLeaderElection(LoadBalancerTest.java:737)

Usage tip: To enable automatic navigation to failure message, open the links with CTRL/CMD-click.
example failure 2021-01-31T13:37:02.0885058Z

Full exception stacktrace 
java.lang.AssertionError: expected [Optional[LeaderBroker(serviceUrl=http://localhost4:36589)]] but found [Optional.empty]
	at org.testng.Assert.fail(Assert.java:99)
	at org.testng.Assert.failNotEquals(Assert.java:1037)
	at org.testng.Assert.assertEqualsImpl(Assert.java:140)
	at org.testng.Assert.assertEquals(Assert.java:122)
	at org.testng.Assert.assertEquals(Assert.java:617)
	at org.apache.pulsar.broker.loadbalance.LoadBalancerTest.testLeaderElection(LoadBalancerTest.java:737)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
	at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:45)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:73)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


java.lang.AssertionError: expected [Optional[LeaderBroker(serviceUrl=http://localhost4:36253)]] but found [Optional.empty]
	at org.testng.Assert.fail(Assert.java:99)
	at org.testng.Assert.failNotEquals(Assert.java:1037)
	at org.testng.Assert.assertEqualsImpl(Assert.java:140)
	at org.testng.Assert.assertEquals(Assert.java:122)
	at org.testng.Assert.assertEquals(Assert.java:617)
	at org.apache.pulsar.broker.loadbalance.LoadBalancerTest.testLeaderElection(LoadBalancerTest.java:737)

Usage tip: To enable automatic navigation to failure message, open the links with CTRL/CMD-click.
example failure 2021-01-27T23:15:52.1427955Z

Full exception stacktrace 
java.lang.AssertionError: expected [Optional[LeaderBroker(serviceUrl=http://localhost4:36253)]] but found [Optional.empty]
	at org.testng.Assert.fail(Assert.java:99)
	at org.testng.Assert.failNotEquals(Assert.java:1037)
	at org.testng.Assert.assertEqualsImpl(Assert.java:140)
	at org.testng.Assert.assertEquals(Assert.java:122)
	at org.testng.Assert.assertEquals(Assert.java:617)
	at org.apache.pulsar.broker.loadbalance.LoadBalancerTest.testLeaderElection(LoadBalancerTest.java:737)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
	at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:45)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:73)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


java.lang.AssertionError: expected [Optional[LeaderBroker(serviceUrl=http://localhost4:46437)]] but found [Optional.empty]
	at org.testng.Assert.fail(Assert.java:99)
	at org.testng.Assert.failNotEquals(Assert.java:1037)
	at org.testng.Assert.assertEqualsImpl(Assert.java:140)
	at org.testng.Assert.assertEquals(Assert.java:122)
	at org.testng.Assert.assertEquals(Assert.java:617)
	at org.apache.pulsar.broker.loadbalance.LoadBalancerTest.testLeaderElection(LoadBalancerTest.java:737)

Usage tip: To enable automatic navigation to failure message, open the links with CTRL/CMD-click.
example failure 2021-01-27T13:20:28.5577038Z

Full exception stacktrace 
java.lang.AssertionError: expected [Optional[LeaderBroker(serviceUrl=http://localhost4:46437)]] but found [Optional.empty]
	at org.testng.Assert.fail(Assert.java:99)
	at org.testng.Assert.failNotEquals(Assert.java:1037)
	at org.testng.Assert.assertEqualsImpl(Assert.java:140)
	at org.testng.Assert.assertEquals(Assert.java:122)
	at org.testng.Assert.assertEquals(Assert.java:617)
	at org.apache.pulsar.broker.loadbalance.LoadBalancerTest.testLeaderElection(LoadBalancerTest.java:737)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
	at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:45)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:73)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


java.lang.AssertionError: expected [Optional[LeaderBroker(serviceUrl=http://localhost4:34401)]] but found [Optional.empty]
	at org.testng.Assert.fail(Assert.java:99)
	at org.testng.Assert.failNotEquals(Assert.java:1037)
	at org.testng.Assert.assertEqualsImpl(Assert.java:140)
	at org.testng.Assert.assertEquals(Assert.java:122)
	at org.testng.Assert.assertEquals(Assert.java:617)
	at org.apache.pulsar.broker.loadbalance.LoadBalancerTest.testLeaderElection(LoadBalancerTest.java:737)

Usage tip: To enable automatic navigation to failure message, open the links with CTRL/CMD-click.
example failure 2021-01-26T21:22:27.4025488Z

Full exception stacktrace 
java.lang.AssertionError: expected [Optional[LeaderBroker(serviceUrl=http://localhost4:34401)]] but found [Optional.empty]
	at org.testng.Assert.fail(Assert.java:99)
	at org.testng.Assert.failNotEquals(Assert.java:1037)
	at org.testng.Assert.assertEqualsImpl(Assert.java:140)
	at org.testng.Assert.assertEquals(Assert.java:122)
	at org.testng.Assert.assertEquals(Assert.java:617)
	at org.apache.pulsar.broker.loadbalance.LoadBalancerTest.testLeaderElection(LoadBalancerTest.java:737)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
	at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:45)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:73)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


java.lang.AssertionError: expected [Optional[LeaderBroker(serviceUrl=http://localhost1:45573)]] but found [Optional.empty]
	at org.testng.Assert.fail(Assert.java:99)
	at org.testng.Assert.failNotEquals(Assert.java:1037)
	at org.testng.Assert.assertEqualsImpl(Assert.java:140)
	at org.testng.Assert.assertEquals(Assert.java:122)
	at org.testng.Assert.assertEquals(Assert.java:617)
	at org.apache.pulsar.broker.loadbalance.LoadBalancerTest.testLeaderElection(LoadBalancerTest.java:737)

Usage tip: To enable automatic navigation to failure message, open the links with CTRL/CMD-click.
example failure 2021-02-01T11:45:44.5009257Z

Full exception stacktrace 
java.lang.AssertionError: expected [Optional[LeaderBroker(serviceUrl=http://localhost1:45573)]] but found [Optional.empty]
	at org.testng.Assert.fail(Assert.java:99)
	at org.testng.Assert.failNotEquals(Assert.java:1037)
	at org.testng.Assert.assertEqualsImpl(Assert.java:140)
	at org.testng.Assert.assertEquals(Assert.java:122)
	at org.testng.Assert.assertEquals(Assert.java:617)
	at org.apache.pulsar.broker.loadbalance.LoadBalancerTest.testLeaderElection(LoadBalancerTest.java:737)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
	at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:45)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:73)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


java.lang.AssertionError: expected [Optional[LeaderBroker(serviceUrl=http://localhost1:43183)]] but found [Optional.empty]
	at org.testng.Assert.fail(Assert.java:99)
	at org.testng.Assert.failNotEquals(Assert.java:1037)
	at org.testng.Assert.assertEqualsImpl(Assert.java:140)
	at org.testng.Assert.assertEquals(Assert.java:122)
	at org.testng.Assert.assertEquals(Assert.java:617)
	at org.apache.pulsar.broker.loadbalance.LoadBalancerTest.testLeaderElection(LoadBalancerTest.java:737)

Usage tip: To enable automatic navigation to failure message, open the links with CTRL/CMD-click.
example failure 2021-02-01T04:07:41.3221135Z

Full exception stacktrace 
java.lang.AssertionError: expected [Optional[LeaderBroker(serviceUrl=http://localhost1:43183)]] but found [Optional.empty]
	at org.testng.Assert.fail(Assert.java:99)
	at org.testng.Assert.failNotEquals(Assert.java:1037)
	at org.testng.Assert.assertEqualsImpl(Assert.java:140)
	at org.testng.Assert.assertEquals(Assert.java:122)
	at org.testng.Assert.assertEquals(Assert.java:617)
	at org.apache.pulsar.broker.loadbalance.LoadBalancerTest.testLeaderElection(LoadBalancerTest.java:737)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
	at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:45)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:73)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


java.lang.AssertionError: expected [Optional[LeaderBroker(serviceUrl=http://localhost3:32955)]] but found [Optional.empty]
	at org.testng.Assert.fail(Assert.java:99)
	at org.testng.Assert.failNotEquals(Assert.java:1037)
	at org.testng.Assert.assertEqualsImpl(Assert.java:140)
	at org.testng.Assert.assertEquals(Assert.java:122)
	at org.testng.Assert.assertEquals(Assert.java:617)
	at org.apache.pulsar.broker.loadbalance.LoadBalancerTest.testLeaderElection(LoadBalancerTest.java:737)

Usage tip: To enable automatic navigation to failure message, open the links with CTRL/CMD-click.
example failure 2021-01-26T21:38:08.1357619Z

Full exception stacktrace 
java.lang.AssertionError: expected [Optional[LeaderBroker(serviceUrl=http://localhost3:32955)]] but found [Optional.empty]
	at org.testng.Assert.fail(Assert.java:99)
	at org.testng.Assert.failNotEquals(Assert.java:1037)
	at org.testng.Assert.assertEqualsImpl(Assert.java:140)
	at org.testng.Assert.assertEquals(Assert.java:122)
	at org.testng.Assert.assertEquals(Assert.java:617)
	at org.apache.pulsar.broker.loadbalance.LoadBalancerTest.testLeaderElection(LoadBalancerTest.java:737)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
	at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:45)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:73)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


java.lang.AssertionError: expected [Optional[LeaderBroker(serviceUrl=http://localhost3:37271)]] but found [Optional.empty]
	at org.testng.Assert.fail(Assert.java:99)
	at org.testng.Assert.failNotEquals(Assert.java:1037)
	at org.testng.Assert.assertEqualsImpl(Assert.java:140)
	at org.testng.Assert.assertEquals(Assert.java:122)
	at org.testng.Assert.assertEquals(Assert.java:617)
	at org.apache.pulsar.broker.loadbalance.LoadBalancerTest.testLeaderElection(LoadBalancerTest.java:737)

Usage tip: To enable automatic navigation to failure message, open the links with CTRL/CMD-click.
example failure 2021-01-27T01:25:39.9608804Z

Full exception stacktrace 
java.lang.AssertionError: expected [Optional[LeaderBroker(serviceUrl=http://localhost3:37271)]] but found [Optional.empty]
	at org.testng.Assert.fail(Assert.java:99)
	at org.testng.Assert.failNotEquals(Assert.java:1037)
	at org.testng.Assert.assertEqualsImpl(Assert.java:140)
	at org.testng.Assert.assertEquals(Assert.java:122)
	at org.testng.Assert.assertEquals(Assert.java:617)
	at org.apache.pulsar.broker.loadbalance.LoadBalancerTest.testLeaderElection(LoadBalancerTest.java:737)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
	at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:45)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:73)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


java.lang.AssertionError: expected [Optional[LeaderBroker(serviceUrl=http://localhost4:38263)]] but found [Optional.empty]
	at org.testng.Assert.fail(Assert.java:99)
	at org.testng.Assert.failNotEquals(Assert.java:1037)
	at org.testng.Assert.assertEqualsImpl(Assert.java:140)
	at org.testng.Assert.assertEquals(Assert.java:122)
	at org.testng.Assert.assertEquals(Assert.java:617)
	at org.apache.pulsar.broker.loadbalance.LoadBalancerTest.testLeaderElection(LoadBalancerTest.java:737)

example failure 2021-01-28T02:07:26.8743907Z

Full exception stacktrace 
java.lang.AssertionError: expected [Optional[LeaderBroker(serviceUrl=http://localhost4:38263)]] but found [Optional.empty]
	at org.testng.Assert.fail(Assert.java:99)
	at org.testng.Assert.failNotEquals(Assert.java:1037)
	at org.testng.Assert.assertEqualsImpl(Assert.java:140)
	at org.testng.Assert.assertEquals(Assert.java:122)
	at org.testng.Assert.assertEquals(Assert.java:617)
	at org.apache.pulsar.broker.loadbalance.LoadBalancerTest.testLeaderElection(LoadBalancerTest.java:737)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
	at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:45)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:73)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


java.lang.AssertionError: expected [Optional[LeaderBroker(serviceUrl=http://localhost1:39203)]] but found [Optional.empty]
	at org.testng.Assert.fail(Assert.java:99)
	at org.testng.Assert.failNotEquals(Assert.java:1037)
	at org.testng.Assert.assertEqualsImpl(Assert.java:140)
	at org.testng.Assert.assertEquals(Assert.java:122)
	at org.testng.Assert.assertEquals(Assert.java:617)
	at org.apache.pulsar.broker.loadbalance.LoadBalancerTest.testLeaderElection(LoadBalancerTest.java:737)

Usage tip: To enable automatic navigation to failure message, open the links with CTRL/CMD-click.
example failure 2021-01-26T21:30:13.3435396Z

Full exception stacktrace 
java.lang.AssertionError: expected [Optional[LeaderBroker(serviceUrl=http://localhost1:39203)]] but found [Optional.empty]
	at org.testng.Assert.fail(Assert.java:99)
	at org.testng.Assert.failNotEquals(Assert.java:1037)
	at org.testng.Assert.assertEqualsImpl(Assert.java:140)
	at org.testng.Assert.assertEquals(Assert.java:122)
	at org.testng.Assert.assertEquals(Assert.java:617)
	at org.apache.pulsar.broker.loadbalance.LoadBalancerTest.testLeaderElection(LoadBalancerTest.java:737)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
	at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:45)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:73)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


java.lang.AssertionError: expected [Optional[LeaderBroker(serviceUrl=http://localhost2:36493)]] but found [Optional.empty]
	at org.testng.Assert.fail(Assert.java:99)
	at org.testng.Assert.failNotEquals(Assert.java:1037)
	at org.testng.Assert.assertEqualsImpl(Assert.java:140)
	at org.testng.Assert.assertEquals(Assert.java:122)
	at org.testng.Assert.assertEquals(Assert.java:617)
	at org.apache.pulsar.broker.loadbalance.LoadBalancerTest.testLeaderElection(LoadBalancerTest.java:737)

Usage tip: To enable automatic navigation to failure message, open the links with CTRL/CMD-click.
example failure 2021-02-01T11:36:57.1174828Z

Full exception stacktrace 
java.lang.AssertionError: expected [Optional[LeaderBroker(serviceUrl=http://localhost2:36493)]] but found [Optional.empty]
	at org.testng.Assert.fail(Assert.java:99)
	at org.testng.Assert.failNotEquals(Assert.java:1037)
	at org.testng.Assert.assertEqualsImpl(Assert.java:140)
	at org.testng.Assert.assertEquals(Assert.java:122)
	at org.testng.Assert.assertEquals(Assert.java:617)
	at org.apache.pulsar.broker.loadbalance.LoadBalancerTest.testLeaderElection(LoadBalancerTest.java:737)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
	at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:45)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:73)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


java.lang.AssertionError: expected [Optional[LeaderBroker(serviceUrl=http://localhost1:46259)]] but found [Optional.empty]
	at org.testng.Assert.fail(Assert.java:99)
	at org.testng.Assert.failNotEquals(Assert.java:1037)
	at org.testng.Assert.assertEqualsImpl(Assert.java:140)
	at org.testng.Assert.assertEquals(Assert.java:122)
	at org.testng.Assert.assertEquals(Assert.java:617)
	at org.apache.pulsar.broker.loadbalance.LoadBalancerTest.testLeaderElection(LoadBalancerTest.java:737)

Usage tip: To enable automatic navigation to failure message, open the links with CTRL/CMD-click.
example failure 2021-01-27T06:49:07.9611925Z

Full exception stacktrace 
java.lang.AssertionError: expected [Optional[LeaderBroker(serviceUrl=http://localhost1:46259)]] but found [Optional.empty]
	at org.testng.Assert.fail(Assert.java:99)
	at org.testng.Assert.failNotEquals(Assert.java:1037)
	at org.testng.Assert.assertEqualsImpl(Assert.java:140)
	at org.testng.Assert.assertEquals(Assert.java:122)
	at org.testng.Assert.assertEquals(Assert.java:617)
	at org.apache.pulsar.broker.loadbalance.LoadBalancerTest.testLeaderElection(LoadBalancerTest.java:737)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
	at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:45)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:73)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


java.lang.AssertionError: expected [Optional[LeaderBroker(serviceUrl=http://localhost4:37953)]] but found [Optional.empty]
	at org.testng.Assert.fail(Assert.java:99)
	at org.testng.Assert.failNotEquals(Assert.java:1037)
	at org.testng.Assert.assertEqualsImpl(Assert.java:140)
	at org.testng.Assert.assertEquals(Assert.java:122)
	at org.testng.Assert.assertEquals(Assert.java:617)
	at org.apache.pulsar.broker.loadbalance.LoadBalancerTest.testLeaderElection(LoadBalancerTest.java:737)

Usage tip: To enable automatic navigation to failure message, open the links with CTRL/CMD-click.
example failure 2021-01-30T09:02:07.6453109Z

Full exception stacktrace 
java.lang.AssertionError: expected [Optional[LeaderBroker(serviceUrl=http://localhost4:37953)]] but found [Optional.empty]
	at org.testng.Assert.fail(Assert.java:99)
	at org.testng.Assert.failNotEquals(Assert.java:1037)
	at org.testng.Assert.assertEqualsImpl(Assert.java:140)
	at org.testng.Assert.assertEquals(Assert.java:122)
	at org.testng.Assert.assertEquals(Assert.java:617)
	at org.apache.pulsar.broker.loadbalance.LoadBalancerTest.testLeaderElection(LoadBalancerTest.java:737)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
	at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:45)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:73)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


java.lang.AssertionError: expected [Optional[LeaderBroker(serviceUrl=http://localhost3:39865)]] but found [Optional.empty]
	at org.testng.Assert.fail(Assert.java:99)
	at org.testng.Assert.failNotEquals(Assert.java:1037)
	at org.testng.Assert.assertEqualsImpl(Assert.java:140)
	at org.testng.Assert.assertEquals(Assert.java:122)
	at org.testng.Assert.assertEquals(Assert.java:617)
	at org.apache.pulsar.broker.loadbalance.LoadBalancerTest.testLeaderElection(LoadBalancerTest.java:737)

Usage tip: To enable automatic navigation to failure message, open the links with CTRL/CMD-click.
example failure 2021-01-27T01:41:04.8537837Z

Full exception stacktrace 
java.lang.AssertionError: expected [Optional[LeaderBroker(serviceUrl=http://localhost3:39865)]] but found [Optional.empty]
	at org.testng.Assert.fail(Assert.java:99)
	at org.testng.Assert.failNotEquals(Assert.java:1037)
	at org.testng.Assert.assertEqualsImpl(Assert.java:140)
	at org.testng.Assert.assertEquals(Assert.java:122)
	at org.testng.Assert.assertEquals(Assert.java:617)
	at org.apache.pulsar.broker.loadbalance.LoadBalancerTest.testLeaderElection(LoadBalancerTest.java:737)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
	at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:45)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:73)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


java.lang.AssertionError: expected [Optional[LeaderBroker(serviceUrl=http://localhost2:34025)]] but found [Optional.empty]
	at org.testng.Assert.fail(Assert.java:99)
	at org.testng.Assert.failNotEquals(Assert.java:1037)
	at org.testng.Assert.assertEqualsImpl(Assert.java:140)
	at org.testng.Assert.assertEquals(Assert.java:122)
	at org.testng.Assert.assertEquals(Assert.java:617)
	at org.apache.pulsar.broker.loadbalance.LoadBalancerTest.testLeaderElection(LoadBalancerTest.java:737)

example failure 2021-02-01T03:57:37.0419371Z

Full exception stacktrace 
java.lang.AssertionError: expected [Optional[LeaderBroker(serviceUrl=http://localhost2:34025)]] but found [Optional.empty]
	at org.testng.Assert.fail(Assert.java:99)
	at org.testng.Assert.failNotEquals(Assert.java:1037)
	at org.testng.Assert.assertEqualsImpl(Assert.java:140)
	at org.testng.Assert.assertEquals(Assert.java:122)
	at org.testng.Assert.assertEquals(Assert.java:617)
	at org.apache.pulsar.broker.loadbalance.LoadBalancerTest.testLeaderElection(LoadBalancerTest.java:737)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
	at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:45)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:73)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


java.lang.AssertionError: expected [Optional[LeaderBroker(serviceUrl=http://localhost4:38887)]] but found [Optional.empty]
	at org.testng.Assert.fail(Assert.java:99)
	at org.testng.Assert.failNotEquals(Assert.java:1037)
	at org.testng.Assert.assertEqualsImpl(Assert.java:140)
	at org.testng.Assert.assertEquals(Assert.java:122)
	at org.testng.Assert.assertEquals(Assert.java:617)
	at org.apache.pulsar.broker.loadbalance.LoadBalancerTest.testLeaderElection(LoadBalancerTest.java:737)

Usage tip: To enable automatic navigation to failure message, open the links with CTRL/CMD-click.
example failure 2021-01-27T20:47:03.8399173Z

Full exception stacktrace 
java.lang.AssertionError: expected [Optional[LeaderBroker(serviceUrl=http://localhost4:38887)]] but found [Optional.empty]
	at org.testng.Assert.fail(Assert.java:99)
	at org.testng.Assert.failNotEquals(Assert.java:1037)
	at org.testng.Assert.assertEqualsImpl(Assert.java:140)
	at org.testng.Assert.assertEquals(Assert.java:122)
	at org.testng.Assert.assertEquals(Assert.java:617)
	at org.apache.pulsar.broker.loadbalance.LoadBalancerTest.testLeaderElection(LoadBalancerTest.java:737)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
	at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:45)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:73)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


java.lang.AssertionError: expected [Optional[LeaderBroker(serviceUrl=http://localhost4:46513)]] but found [Optional.empty]
	at org.testng.Assert.fail(Assert.java:99)
	at org.testng.Assert.failNotEquals(Assert.java:1037)
	at org.testng.Assert.assertEqualsImpl(Assert.java:140)
	at org.testng.Assert.assertEquals(Assert.java:122)
	at org.testng.Assert.assertEquals(Assert.java:617)
	at org.apache.pulsar.broker.loadbalance.LoadBalancerTest.testLeaderElection(LoadBalancerTest.java:737)

example failure 2021-02-01T13:18:35.4373409Z

Full exception stacktrace 
java.lang.AssertionError: expected [Optional[LeaderBroker(serviceUrl=http://localhost4:46513)]] but found [Optional.empty]
	at org.testng.Assert.fail(Assert.java:99)
	at org.testng.Assert.failNotEquals(Assert.java:1037)
	at org.testng.Assert.assertEqualsImpl(Assert.java:140)
	at org.testng.Assert.assertEquals(Assert.java:122)
	at org.testng.Assert.assertEquals(Assert.java:617)
	at org.apache.pulsar.broker.loadbalance.LoadBalancerTest.testLeaderElection(LoadBalancerTest.java:737)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
	at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:45)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:73)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


java.lang.AssertionError: expected [Optional[LeaderBroker(serviceUrl=http://localhost4:40269)]] but found [Optional.empty]
	at org.testng.Assert.fail(Assert.java:99)
	at org.testng.Assert.failNotEquals(Assert.java:1037)
	at org.testng.Assert.assertEqualsImpl(Assert.java:140)
	at org.testng.Assert.assertEquals(Assert.java:122)
	at org.testng.Assert.assertEquals(Assert.java:617)
	at org.apache.pulsar.broker.loadbalance.LoadBalancerTest.testLeaderElection(LoadBalancerTest.java:737)

example failure 2021-01-29T00:19:52.4940690Z

Full exception stacktrace 
java.lang.AssertionError: expected [Optional[LeaderBroker(serviceUrl=http://localhost4:40269)]] but found [Optional.empty]
	at org.testng.Assert.fail(Assert.java:99)
	at org.testng.Assert.failNotEquals(Assert.java:1037)
	at org.testng.Assert.assertEqualsImpl(Assert.java:140)
	at org.testng.Assert.assertEquals(Assert.java:122)
	at org.testng.Assert.assertEquals(Assert.java:617)
	at org.apache.pulsar.broker.loadbalance.LoadBalancerTest.testLeaderElection(LoadBalancerTest.java:737)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
	at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:45)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:73)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


java.lang.AssertionError: expected [Optional[LeaderBroker(serviceUrl=http://localhost4:41803)]] but found [Optional.empty]
	at org.testng.Assert.fail(Assert.java:99)
	at org.testng.Assert.failNotEquals(Assert.java:1037)
	at org.testng.Assert.assertEqualsImpl(Assert.java:140)
	at org.testng.Assert.assertEquals(Assert.java:122)
	at org.testng.Assert.assertEquals(Assert.java:617)
	at org.apache.pulsar.broker.loadbalance.LoadBalancerTest.testLeaderElection(LoadBalancerTest.java:737)

example failure 2021-01-27T23:30:22.3669629Z

Full exception stacktrace 
java.lang.AssertionError: expected [Optional[LeaderBroker(serviceUrl=http://localhost4:41803)]] but found [Optional.empty]
	at org.testng.Assert.fail(Assert.java:99)
	at org.testng.Assert.failNotEquals(Assert.java:1037)
	at org.testng.Assert.assertEqualsImpl(Assert.java:140)
	at org.testng.Assert.assertEquals(Assert.java:122)
	at org.testng.Assert.assertEquals(Assert.java:617)
	at org.apache.pulsar.broker.loadbalance.LoadBalancerTest.testLeaderElection(LoadBalancerTest.java:737)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
	at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:45)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:73)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


java.lang.AssertionError: expected [Optional[LeaderBroker(serviceUrl=http://localhost1:42221)]] but found [Optional.empty]
	at org.testng.Assert.fail(Assert.java:99)
	at org.testng.Assert.failNotEquals(Assert.java:1037)
	at org.testng.Assert.assertEqualsImpl(Assert.java:140)
	at org.testng.Assert.assertEquals(Assert.java:122)
	at org.testng.Assert.assertEquals(Assert.java:617)
	at org.apache.pulsar.broker.loadbalance.LoadBalancerTest.testLeaderElection(LoadBalancerTest.java:737)

example failure 2021-01-27T06:40:00.5182280Z

Full exception stacktrace 
java.lang.AssertionError: expected [Optional[LeaderBroker(serviceUrl=http://localhost1:42221)]] but found [Optional.empty]
	at org.testng.Assert.fail(Assert.java:99)
	at org.testng.Assert.failNotEquals(Assert.java:1037)
	at org.testng.Assert.assertEqualsImpl(Assert.java:140)
	at org.testng.Assert.assertEquals(Assert.java:122)
	at org.testng.Assert.assertEquals(Assert.java:617)
	at org.apache.pulsar.broker.loadbalance.LoadBalancerTest.testLeaderElection(LoadBalancerTest.java:737)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
	at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:45)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:73)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


java.util.NoSuchElementException: No value present
	at java.util.Optional.get(Optional.java:135)
	at org.apache.pulsar.broker.loadbalance.LoadBalancerTest.loopUntilLeaderChanges(LoadBalancerTest.java:186)
	at org.apache.pulsar.broker.loadbalance.LoadBalancerTest.testLeaderElection(LoadBalancerTest.java:744)

example failure 2021-01-28T18:29:27.4273404Z

Full exception stacktrace 
java.util.NoSuchElementException: No value present
	at java.util.Optional.get(Optional.java:135)
	at org.apache.pulsar.broker.loadbalance.LoadBalancerTest.loopUntilLeaderChanges(LoadBalancerTest.java:186)
	at org.apache.pulsar.broker.loadbalance.LoadBalancerTest.testLeaderElection(LoadBalancerTest.java:744)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
	at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:45)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:73)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


java.lang.AssertionError: expected [Optional[LeaderBroker(serviceUrl=http://localhost1:41757)]] but found [Optional.empty]
	at org.testng.Assert.fail(Assert.java:99)
	at org.testng.Assert.failNotEquals(Assert.java:1037)
	at org.testng.Assert.assertEqualsImpl(Assert.java:140)
	at org.testng.Assert.assertEquals(Assert.java:122)
	at org.testng.Assert.assertEquals(Assert.java:617)
	at org.apache.pulsar.broker.loadbalance.LoadBalancerTest.testLeaderElection(LoadBalancerTest.java:737)

example failure 2021-01-27T06:29:47.3585099Z

Full exception stacktrace 
java.lang.AssertionError: expected [Optional[LeaderBroker(serviceUrl=http://localhost1:41757)]] but found [Optional.empty]
	at org.testng.Assert.fail(Assert.java:99)
	at org.testng.Assert.failNotEquals(Assert.java:1037)
	at org.testng.Assert.assertEqualsImpl(Assert.java:140)
	at org.testng.Assert.assertEquals(Assert.java:122)
	at org.testng.Assert.assertEquals(Assert.java:617)
	at org.apache.pulsar.broker.loadbalance.LoadBalancerTest.testLeaderElection(LoadBalancerTest.java:737)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
	at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:45)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:73)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


java.lang.AssertionError: expected [Optional[LeaderBroker(serviceUrl=http://localhost4:37805)]] but found [Optional.empty]
	at org.testng.Assert.fail(Assert.java:99)
	at org.testng.Assert.failNotEquals(Assert.java:1037)
	at org.testng.Assert.assertEqualsImpl(Assert.java:140)
	at org.testng.Assert.assertEquals(Assert.java:122)
	at org.testng.Assert.assertEquals(Assert.java:617)
	at org.apache.pulsar.broker.loadbalance.LoadBalancerTest.testLeaderElection(LoadBalancerTest.java:737)

example failure 2021-01-27T08:05:03.5029714Z

Full exception stacktrace 
java.lang.AssertionError: expected [Optional[LeaderBroker(serviceUrl=http://localhost4:37805)]] but found [Optional.empty]
	at org.testng.Assert.fail(Assert.java:99)
	at org.testng.Assert.failNotEquals(Assert.java:1037)
	at org.testng.Assert.assertEqualsImpl(Assert.java:140)
	at org.testng.Assert.assertEquals(Assert.java:122)
	at org.testng.Assert.assertEquals(Assert.java:617)
	at org.apache.pulsar.broker.loadbalance.LoadBalancerTest.testLeaderElection(LoadBalancerTest.java:737)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
	at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:45)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:73)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


java.lang.AssertionError: expected [Optional[LeaderBroker(serviceUrl=http://localhost4:37205)]] but found [Optional.empty]
	at org.testng.Assert.fail(Assert.java:99)
	at org.testng.Assert.failNotEquals(Assert.java:1037)
	at org.testng.Assert.assertEqualsImpl(Assert.java:140)
	at org.testng.Assert.assertEquals(Assert.java:122)
	at org.testng.Assert.assertEquals(Assert.java:617)
	at org.apache.pulsar.broker.loadbalance.LoadBalancerTest.testLeaderElection(LoadBalancerTest.java:737)

example failure 2021-01-27T21:50:29.0310289Z

Full exception stacktrace 
java.lang.AssertionError: expected [Optional[LeaderBroker(serviceUrl=http://localhost4:37205)]] but found [Optional.empty]
	at org.testng.Assert.fail(Assert.java:99)
	at org.testng.Assert.failNotEquals(Assert.java:1037)
	at org.testng.Assert.assertEqualsImpl(Assert.java:140)
	at org.testng.Assert.assertEquals(Assert.java:122)
	at org.testng.Assert.assertEquals(Assert.java:617)
	at org.apache.pulsar.broker.loadbalance.LoadBalancerTest.testLeaderElection(LoadBalancerTest.java:737)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
	at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:45)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:73)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Activity

  1. lhotari
    added
    type/bugThe PR fixed a bug or issue reported a bug
    on Feb 1, 2021
  2. jiazhai
    added
    type/flaky-tests
    help wanted
    on Feb 2, 2021

  3. lhotari commented on Feb 2, 2021

    @lhotari
    lhotari
    on Feb 2, 2021
    MemberAuthor

    I have started work on fixing this flaky test.

  4. lhotari commented on Feb 2, 2021

    @lhotari
    lhotari
    on Feb 2, 2021
    MemberAuthor

    I was able to reproduce the flakiness and I made some changes in master...lhotari:lh-fix-flaky-load-balancer-test . However the flakiness doesn't go away.
    The logs are in https://gist.github.com/lhotari/0653dc5d9ca418a1bca2d01a632fd655 .

    The problem is that the leader doesn't get assigned to one of the follower brokers. The value of the current leaders stays Optional.empty. This can be found in the logs around these lines:

    10:38:34.605 [TestNG-method=testLeaderElection-1] INFO  org.apache.pulsar.broker.loadbalance.LoadBalancerTest - New leader is : http://localhost4:32957
10:38:34.605 [TestNG-method=testLeaderElection-1] INFO  org.apache.pulsar.broker.loadbalance.LoadBalancerTest - Old leader is : http://localhost4:32957
10:38:34.705 [awaitility-thread] INFO  org.apache.pulsar.broker.loadbalance.LoadBalancerTest - Current leader for http://localhost4:32957 is : Optional[LeaderBroker(serviceUrl=http://localhost4:32957)]
10:38:34.806 [awaitility-thread] INFO  org.apache.pulsar.broker.loadbalance.LoadBalancerTest - Current leader for http://localhost2:35915 is : Optional[LeaderBroker(serviceUrl=http://localhost4:32957)]
10:38:34.906 [awaitility-thread] INFO  org.apache.pulsar.broker.loadbalance.LoadBalancerTest - Current leader for http://localhost1:43075 is : Optional.empty
10:38:35.008 [awaitility-thread] INFO  org.apache.pulsar.broker.loadbalance.LoadBalancerTest - Current leader for http://localhost1:43075 is : Optional.empty
10:38:35.108 [awaitility-thread] INFO  org.apache.pulsar.broker.loadbalance.LoadBalancerTest - Current leader for http://localhost1:43075 is : Optional.empty
10:38:35.208 [awaitility-thread] INFO  org.apache.pulsar.broker.loadbalance.LoadBalancerTest - Current leader for http://localhost1:43075 is : Optional.empty
10:38:35.308 [awaitility-thread] INFO  org.apache.pulsar.broker.loadbalance.LoadBalancerTest - Current leader for http://localhost1:43075 is : Optional.empty
10:38:35.408 [awaitility-thread] INFO  org.apache.pulsar.broker.loadbalance.LoadBalancerTest - Current leader for http://localhost1:43075 is : Optional.empty
10:38:35.509 [awaitility-thread] INFO  org.apache.pulsar.broker.loadbalance.LoadBalancerTest - Current leader for http://localhost1:43075 is : Optional.empty
10:38:35.609 [awaitility-thread] INFO  org.apache.pulsar.broker.loadbalance.LoadBalancerTest - Current leader for http://localhost1:43075 is : Optional.empty
10:38:35.709 [awaitility-thread] INFO  org.apache.pulsar.broker.loadbalance.LoadBalancerTest - Current leader for http://localhost1:43075 is : Optional.empty
10:38:35.809 [awaitility-thread] INFO  org.apache.pulsar.broker.loadbalance.LoadBalancerTest - Current leader for http://localhost1:43075 is : Optional.empty
10:38:35.910 [awaitility-thread] INFO  org.apache.pulsar.broker.loadbalance.LoadBalancerTest - Current leader for http://localhost1:43075 is : Optional.empty
10:38:36.010 [awaitility-thread] INFO  org.apache.pulsar.broker.loadbalance.LoadBalancerTest - Current leader for http://localhost1:43075 is : Optional.empty
10:38:36.110 [awaitility-thread] INFO  org.apache.pulsar.broker.loadbalance.LoadBalancerTest - Current leader for http://localhost1:43075 is : Optional.empty
10:38:36.210 [awaitility-thread] INFO  org.apache.pulsar.broker.loadbalance.LoadBalancerTest - Current leader for http://localhost1:43075 is : Optional.empty
10:38:36.310 [awaitility-thread] INFO  org.apache.pulsar.broker.loadbalance.LoadBalancerTest - Current leader for http://localhost1:43075 is : Optional.empty
10:38:36.411 [awaitility-thread] INFO  org.apache.pulsar.broker.loadbalance.LoadBalancerTest - Current leader for http://localhost1:43075 is : Optional.empty
10:38:36.511 [awaitility-thread] INFO  org.apache.pulsar.broker.loadbalance.LoadBalancerTest - Current leader for http://localhost1:43075 is : Optional.empty
10:38:36.611 [awaitility-thread] INFO  org.apache.pulsar.broker.loadbalance.LoadBalancerTest - Current leader for http://localhost1:43075 is : Optional.empty
10:38:36.711 [awaitility-thread] INFO  org.apache.pulsar.broker.loadbalance.LoadBalancerTest - Current leader for http://localhost1:43075 is : Optional.empty
10:38:36.811 [awaitility-thread] INFO  org.apache.pulsar.broker.loadbalance.LoadBalancerTest - Current leader for http://localhost1:43075 is : Optional.empty
10:38:36.912 [awaitility-thread] INFO  org.apache.pulsar.broker.loadbalance.LoadBalancerTest - Current leader for http://localhost1:43075 is : Optional.empty
... (continue until it times out)

  5. lhotari commented on Feb 2, 2021

    @lhotari
    lhotari
    on Feb 2, 2021
    MemberAuthor

    This was the stacktrace for the NPE that I happened to come across. I wonder if this is a corner case at startup time. The NPE happened in a log statement. I just wonder why the stack is so deep.

    09:43:34.650 [metadata-store-1722-1] WARN  org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl - Exception in state change listener
java.lang.NullPointerException: null
	at org.apache.pulsar.broker.PulsarService.lambda$startLeaderElectionService$3(PulsarService.java:731) ~[classes/:?]
	at org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl.lambda$accept$12(LeaderElectionImpl.java:257) ~[pulsar-metadata-2.8.0-SNAPSHOT.jar:2.8.0-SNAPSHOT]
	at java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:884) ~[?:1.8.0_275]
	at java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:866) ~[?:1.8.0_275]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[?:1.8.0_275]
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) ~[?:1.8.0_275]
	at org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl.lambda$null$9(LeaderElectionImpl.java:177) ~[pulsar-metadata-2.8.0-SNAPSHOT.jar:2.8.0-SNAPSHOT]
	at java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:884) ~[?:1.8.0_275]
	at java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:866) ~[?:1.8.0_275]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[?:1.8.0_275]
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) ~[?:1.8.0_275]
	at org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl.lambda$null$9(LeaderElectionImpl.java:177) ~[pulsar-metadata-2.8.0-SNAPSHOT.jar:2.8.0-SNAPSHOT]
	at java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:884) ~[?:1.8.0_275]
	at java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:866) ~[?:1.8.0_275]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[?:1.8.0_275]
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) ~[?:1.8.0_275]
	at org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl.lambda$null$9(LeaderElectionImpl.java:177) ~[pulsar-metadata-2.8.0-SNAPSHOT.jar:2.8.0-SNAPSHOT]
	at java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:884) ~[?:1.8.0_275]
	at java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:866) ~[?:1.8.0_275]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[?:1.8.0_275]
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) ~[?:1.8.0_275]
	at org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl.lambda$null$9(LeaderElectionImpl.java:177) ~[pulsar-metadata-2.8.0-SNAPSHOT.jar:2.8.0-SNAPSHOT]
	at java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:884) ~[?:1.8.0_275]
	at java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:866) ~[?:1.8.0_275]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[?:1.8.0_275]
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) ~[?:1.8.0_275]
	at org.apache.pulsar.metadata.coordination.impl.LeaderElectionImpl.lambda$null$9(LeaderElectionImpl.java:177) ~[pulsar-metadata-2.8.0-SNAPSHOT.jar:2.8.0-SNAPSHOT]
	at java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:884) ~[?:1.8.0_275]
	at java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:866) ~[?:1.8.0_275]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[?:1.8.0_275]
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) ~[?:1.8.0_275]
.
.
.
THIS KEEPS GOING ON and ON until 1024 stackframes
.
.
.

    full stacktrace which seems to be truncated to 1024 stack frames: https://gist.github.com/lhotari/11cbf31cfad7d82185f42e8f06e2b5a2

  6. lhotari commented on Feb 2, 2021

    @lhotari
    lhotari
    on Feb 2, 2021
    MemberAuthor

    @merlimat Would you mind taking a look at the comments in this issue? It seems that the flakiness might be related to recent PIP-45 changes made in #9240.

    The stack trace with 1024 stack frames seems concerning. The same lines seem to repeat there, looking like a very deep recursion. Is there an explanation for this?

  7. merlimat
    self-assigned this
    on Feb 3, 2021
  8. merlimat
    added this to the 2.8.0 milestone on Feb 3, 2021

  9. merlimat commented on Feb 3, 2021

    @merlimat
    merlimat
    on Feb 3, 2021
    Contributor

    @lhotari Submitted a fix at #9460

  10. lhotari commented on Feb 4, 2021

    @lhotari
    lhotari
    on Feb 4, 2021
    MemberAuthor

    Submitted a fix at #9460

    Thank you @merlimat

  11. merlimat
    closed this as completedin #9443on Feb 6, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

help wantedtype/bugThe PR fixed a bug or issue reported a bugtype/flaky-tests

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions