We had performance issues with a three-server farm (WFE, APP and IDX), so we decided to create a new farm with load balancing (two WFE for user requests, one WFE for serving search requests, one app and one index server).
So far so good. The application performed better, but once in a while it was very slow. Thus with developer dashboard we noticed that http://tempuri.org/ISearchQueryServiceApplication/Execute was taking 15 – 30 seconds to execute! We have some webparts on the page which are firing a FullTextSqlQuery to get results and these webparts were slow. After a refresh the same operation took less than 0.1 second.
So we thought: SQL should be the bottleneck, because the SQL-query is slow. Thus, we ran defragements, shrinks, reorganize-scripts, SQL Profiler, Performance monitors, checked for locks; we followed this article: http://support.microsoft.com/kb/298475 (although for 2005, applicable for 2008 R2). However, no luck, SQL was performing well.
Then we checked the health analyzer in SharePoint, like missing server side dependencies. We also noticed that the content database should be upgraded, almost a hurray came out of our mouths, but no luck. We fixed almost all errors in the server side dependencies using these links: http://spandps.com/2012/01/17/missing-server-side-dependenciesuseful-links-for-diagnosis-and-removal-sp2010-ps2010-sharepoint-projectserver-in/
Still, no luck. Then we thought the problem could be the app pool of the searchservice resetting. We checked all settings in the SharePoint Web Service, like application pool time-outs. These settings were default, so this should not be the problem.
Finally, another Google search and we came up to this post: http://sharepoint.stackexchange.com/questions/23502/sharepoint-takes-15-seconds-to-load-page . Hurray, a very similar problem, because our servers also do not have internet access! We tried the fix in the support article. Hurray, the problem was solved!
Summary: because we moved to a load balanced farm, the servers had to communicate more with each other, but a SSL-check randomly caused for problems, because the servers have no internet access.
