Service Bus Gateway service stuck at Starting

Lesson Learned: Never leave your servers to automatically download and install the updates.

We had a SharePoint farm where Workflow manager is configured on one of its server and rest of them are just using WFM client to connect with the server. Everything was running smooth until some updates got installed on the servers over night and the next morning Workflows stuck and start giving errors.

As initial check, I verified SharePoint services and WFM services, noticed that the Service Bus Gateway service is struck at Starting state. This was very generic thing but at least a point to start thinking...

Troubleshooting:

1. Tried manual restart, but no luck.

2. Tried Leaving and Joining the WFM Farm again. But the timeout error.

Error message received during rejoining: 

“Starting Service bus gateway on

machine <servername> failed: timeout has expired and the operation has not been completed”

3. Tried ULS check and noticed below error.

" 

Service cannot be started. Microsoft.ServiceBus.Commands.Common.Exceptions.HostNotConfiguredException: Entry for host XYZAPPServer is not present in Service Bus farm management database. ---> System.Data.SqlClient.SqlException: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server) ---> System.ComponentModel.Win32Exception: The network path was not found

   --- End of inner exception stack trace ---

   at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)

   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)

   at System.Data.SqlClient.TdsParser.Connect(ServerInfo ser...

"

4. Workflow Manager site was also found blank and empty, this was a major issue that could be causing the Service Bus Gateway to fail. Tried to start the web but faced below error.

Problem Found:

We were on the edge of redoing the WFM installation when something hit in my mind and I checked the installed updates on the server. I noticed that the recently installed updates for Service Bus Manager updated the version to 1.1. This below update is an update to a security fix in Service Bus about DOS. Check more details on below links about the patch update. 

https://www.microsoft.com/en-pk/download/details.aspx?id=43628

https://support.microsoft.com/en-us/topic/ms14-042-vulnerability-in-microsoft-service-bus-could-allow-denial-of-service-july-8-2014-8fd640cb-91fa-87d3-2586-ba19c3f4e592

https://docs.microsoft.com/en-us/security-updates/SecurityBulletins/2014/ms14-042?redirectedfrom=MSDN

Resolution:

To resolve this issue, first Leave the WFM farm. After that, we uninstalled the above mentioned updates and install the correct patch and update for Service Bus from the below mentioned location. Install Update for Service Bus Server 1.1 (KB3086798) .

https://www.microsoft.com/en-us/download/details.aspx?id=49496

Once the installation is complete, join the WFM farm again using the wizard and register with your web application.

Register WFM to SharePoint. Open SharePoint Management Shell and run below command according to your site and Workflow Host URL.

Register-SPWorkflowService –SPSite https://yourSite/ –WorkflowHostUri https://SampleServer:12290/ -AllowOAuthhttp -force.

Note: Use 12291 port if you are using HTTPS.

Hope it will help someone. :)

Reference: https://docs.microsoft.com/en-us/archive/blogs/shashi/service-bus-gateway-service-struck-in-starting-state