Dolphinscheduler schedules and generates workflow instances. No task instances are generated. Solution process one result (or null) to be returned by selectOne(), but found: 2

background:
The problem occurred during online scheduling and when manually starting tasks, the master reported the following error: Several tasks were found to be invalid.

[ERROR] 2022-08-01 09:59:57.988 org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteThread:[244] - handler error:
org.mybatis.spring.MyBatisSystemException: nested exception is org.apache.ibatis.exceptions.TooManyResultsException: Expected one result (or null) to be returned by selectOne(), but found: 2
at org.mybatis.spring.MyBatisExceptionTranslator.translateExceptionIfPossible(MyBatisExceptionTranslator.java:78)
at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:440)
at com.sun.proxy.$Proxy91.selectOne(Unknown Source)
at org.mybatis.spring.SqlSessionTemplate.selectOne(SqlSessionTemplate.java:159)
at com.baomidou.mybatisplus.core.override.MybatisMapperMethod.execute(MybatisMapperMethod.java:89)
at com.baomidou.mybatisplus.core.override.MybatisMapperProxy.invoke(MybatisMapperProxy.java:61)
at com.sun.proxy.$Proxy115.queryByDefinitionCodeAndVersion(Unknown Source)
at sun.reflect.GeneratedMethodAccessor184.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.cache.interceptor.CacheInterceptor.lambda$invoke$0(CacheInterceptor.java:54)
at org.springframework.cache.interceptor.CacheAspectSupport.invokeOperation(CacheAspectSupport.java:366)
at org.springframework.cache.interceptor.CacheAspectSupport.lambda$handleSynchronizedGet$1(CacheAspectSupport.java:447)
at org.springframework.cache.support.NoOpCache.get(NoOpCache.java:76)
at org.springframework.cache.interceptor.CacheAspectSupport.handleSynchronizedGet(CacheAspectSupport.java:442)
at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:382)
at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:345)
at org.springframework.cache.interceptor.CacheInterceptor.invoke(CacheInterceptor.java:64)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy116.queryByDefinitionCodeAndVersion(Unknown Source)
at org.apache.dolphinscheduler.service.process.ProcessService.findTaskDefinition(ProcessService.java:2474)
at org.apache.dolphinscheduler.service.process.ProcessService.lambda$getTaskDefineLogListByRelation$4(ProcessService.java:2465)
at java.util.HashMap.forEach(HashMap.java:1289)
at org.apache.dolphinscheduler.service.process.ProcessService.getTaskDefineLogListByRelation(ProcessService.java:2464)
at org.apache.dolphinscheduler.service.process.ProcessService$$FastClassBySpringCGLIB$$ed138739.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:689)
at org.apache.dolphinscheduler.service.process.ProcessService$$EnhancerBySpringCGLIB$$315b33b2.getTaskDefineLogListByRelation(<generated>)
at org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteThread.buildFlowDag(WorkflowExecuteThread.java:578)
at org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteThread.startProcess(WorkflowExecuteThread.java:542)
at org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteThread.run(WorkflowExecuteThread.java:239)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.ibatis.exceptions.TooManyResultsException: Expected one result (or null) to be returned by selectOne(), but found: 2
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectOne(DefaultSqlSession.java:80)
at sun.reflect.GeneratedMethodAccessor114.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:426)
... 42 common frames omitted

Phenomenon
When I went online to complete the manual running task, I found that the workflow instance started running, but the task instance never started.

Looking at the master log, you can see the following error:

[ERROR] 2022-08-05 10:48:30.764 org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteThread:[244] - handler error:
org.mybatis.spring.MyBatisSystemException: nested exception is org.apache.ibatis.exceptions.TooManyResultsException: Expected one result (or null) to be returned by selectOne(), but found: 2
at org.mybatis.spring.MyBatisExceptionTranslator.translateExceptionIfPossible(MyBatisExceptionTranslator.java:78)
at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:440)
at com.sun.proxy.$Proxy91.selectOne(Unknown Source)
at org.mybatis.spring.SqlSessionTemplate.selectOne(SqlSessionTemplate.java:159)
at com.baomidou.mybatisplus.core.override.MybatisMapperMethod.execute(MybatisMapperMethod.java:89)
at com.baomidou.mybatisplus.core.override.MybatisMapperProxy.invoke(MybatisMapperProxy.java:61)
at com.sun.proxy.$Proxy115.queryByDefinitionCodeAndVersion(Unknown Source)
at sun.reflect.GeneratedMethodAccessor246.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.cache.interceptor.CacheInterceptor.lambda$invoke$0(CacheInterceptor.java:54)
at org.springframework.cache.interceptor.CacheAspectSupport.invokeOperation(CacheAspectSupport.java:366)
at org.springframework.cache.interceptor.CacheAspectSupport.lambda$handleSynchronizedGet$1(CacheAspectSupport.java:447)
at org.springframework.cache.support.NoOpCache.get(NoOpCache.java:76)
at org.springframework.cache.interceptor.CacheAspectSupport.handleSynchronizedGet(CacheAspectSupport.java:442)
at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:382)
at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:345)
at org.springframework.cache.interceptor.CacheInterceptor.invoke(CacheInterceptor.java:64)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy116.queryByDefinitionCodeAndVersion(Unknown Source)
at org.apache.dolphinscheduler.service.process.ProcessService.findTaskDefinition(ProcessService.java:2474)
at org.apache.dolphinscheduler.service.process.ProcessService.lambda$getTaskDefineLogListByRelation$4(ProcessService.java:2465)
at java.util.HashMap.forEach(HashMap.java:1289)
at org.apache.dolphinscheduler.service.process.ProcessService.getTaskDefineLogListByRelation(ProcessService.java:2464)
at org.apache.dolphinscheduler.service.process.ProcessService$$FastClassBySpringCGLIB$$ed138739.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:689)
at org.apache.dolphinscheduler.service.process.ProcessService$$EnhancerBySpringCGLIB$$bf03c416.getTaskDefineLogListByRelation(<generated>)
at org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteThread.buildFlowDag(WorkflowExecuteThread.java:578)
at org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteThread.startProcess(WorkflowExecuteThread.java:542)
at org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteThread.run(WorkflowExecuteThread.java:239)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.ibatis.exceptions.TooManyResultsException: Expected one result (or null) to be returned by selectOne(), but found: 2
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectOne(DefaultSqlSession.java:80)
at sun.reflect.GeneratedMethodAccessor79.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:426)
... 42 common frames omitted

It is found that the query data mainly comes from t_ds_task_definition_log and t_ds_task_definition
So we execute

SELECT code ,version,COUNT(*) cnt from t_ds_task_definition_log group by code ,version order by cnt desc

5715108678048,6353133939232,6234790474016,6344296875552,6168625362464,6430246445344

Scheme:

Checking the official website, we found that this is actually a pseudo-distributed bug of dolphinscheduler, which is related to database transactions when writing a library.

The official said that the higher version has been repaired. Considering the cost issue, we are not upgrading the cluster or version. We have adopted a relatively direct method to delete the old data in the duplicate data.

Operation

select * from t_ds_task_definition_log where code = 6430246445344;

delete from t_ds_task_definition_log where id = 419;

t_ds_task_definition data processing

select * from t_ds_task_definition where code = 6430246445344;

delete from t_ds_task_definition_log where id = 275;