In my last post on BizTalk Implementation & Tuning Best Practices I mentioned a threading issue I was trying to resolve. I’m happy to say it’s been resolved! Below are the details and solution.
I have a receive port set up to receive an envelope containing multiple messages. Each message in the envelope instantiates a new instance of an orchestration that does various processing. The processing we’re concerned with here is an Execute shape that calls an external method in a custom .NET library. Again, not to bog things down with too many details, this method does a few things we don’t really care about right now. What we do care about is that it calls a web service and waits for a response x number of times for each method call (x = 1 infinity.)
At the client where this process is being used, the BizTalk server is state-of-the-art, whereas the web server hosting the web service is pretty much a PC from eBay.
Essentially, what we were running into was a situation where the web server could not process requests fast enough and the BizTalk server was failing instances of the orchestration due to a lack of threads in the thread pool. Imagine (since I’m starting to think more and more about golf season!) you’re standing at the ball machine at the driving range trying to get a bucket to practice your slice. There are two dispensers and a few guys in line to buy some balls. If you’re a golfer, you know the ball machine can sometimes take its time in doing its job. Imagine the guy in front of you puts his tokens into the machine, but nothing happens. Eventually, someone is going to be impatient enough to either leave or go find someone to fix it.
Well, BizTalk is that impatient fellow. The web server was unable to return threads fast enough for BizTalk, so BizTalk said enough is enough and failed the instance.
We ran into this before on the same hardware, but in a different situation. For that solution, we modified the process to handle requests in serial instead of in parallel. Not ideal, but it got the job done.
When it came up again I decided I’d better find a more permanent solution.
At first, my internet searching came up with info on similar issues seen through ASP.NET web service calls. There were various articles on the same issue and resolution of the issue, but the two I found most valuable were:
- A post by Tess on web service calls taking forever and how to discover the cause
- A knowledge base article on Contention, poor performance, and deadlocks when you make Web service requests from ASP.NET applications
The solution boils down to modifying the machine.config threading values appropriately to better tune the servers thread pool for web service requests.
Basically, the machine.config doesn’t affect BizTalk. So this didn’t do anything. However, it’s the basis for the permanent solution (read ahead to Solution 3 if the suspense is killing you!)
Through some more googling, I found some references to folks modifying the LowWatermark and HighWatermark settings in BizTalk. These settings are used to set the outbound processing rate for messages. When BizTalk processes more messages than specified in the HighWatermark value, it stops processing new messages until the number of active messages drops below the LowWatermark.
Sounds like this is a possibility, so I gave it a shot. On my dev box I had to lower the HighWatermark to 6 and the LowWatermark to 3 to eliminate the threading errors. Good news/bad news situation. It eliminated the thread pool errors, but it slows down the processing to a snails pace. This isn’t really that big of a deal in this implementation as processing speed is not a factor. So this is what I’m going with…all ready to make the changes to production to fix the issue…until…
I attended an MSDN webcast (referred to in my BizTalk Implementation & Tuning post mentioned earlier) in which I went out on a limb and decided to ask the presenter if he had any input on tracking down the cause of this issue. Low and behold, he did! “He” being Petr Kratochvil of Microsoft. His answer at the time was that there were some CLR Hosting Keys in the registry that could be used to set things like MaxWorkerThreads, MaxIOThreads and MinWorkerThreads specifically for BizTalk. Remember Solution 1?? Machine.config doesn’t control these settings for BizTalk, but these registry keys do!
I did quite a bit of searching on MSDN and google at Petr’s advice, but no luck. I didn’t find anything relating to what he was talking about. So I gave in and sent him an email asking him to elaborate. He very kindly replied with an excerpt from a document the BizTalk Rangers are working on. His reply contained exactly the info I needed on where to find these keys in the registry. Once I created the keys and restarted the BizTalk service, all was well and I’d found my permanent solution!
I promised him I wouldn’t post the content he sent me so as not to steal any of their thunder, but he said they should be posting the content within a few days. Once they post it, I’ll be sure to link to it here.
In the meantime, I did a search on technorati (which, apparently, I should’ve done in the first place!!) and discovered a post by Kevin Smith on Controlling the CLR Thread pool hosted by BizTalk which pretty much details what I had to do. I wish I’d searched the blogs earlier!