Automatically diagnose IIS and ASP.NET website hangs

Automatically diagnose IIS and ASP.NET website hangs(blog.leansentry.com)

41 points by marvinli 13 years ago | 11 comments

bigdubs 13 years ago |

This is pretty awesome for some types of hangs. I would bet most hangs developers experience are because of locking on SessionState. IReadOnlySessionState is your friend.

darrenkopp 13 years ago | |

I've run into this a lot and ended up changing a lot of the internals of how session state works and have _mostly_ eliminated the issue for us. Granted, you still run into it if you have 2 pages running at the same time, but we see it where sometimes it just doesn't release the state (and it was happening fairly frequently)

http://darrenkopp.com/posts/2013/04/10/Playing-with-fire-Opt...

mvolo 13 years ago | |

hey bigdubs,

That's certainly a good one! One of the things we'll do to help resolve the hang is to identify where blocking is happening. This will pinpoint the session state module in AcquireRequestState as the culprit.

Although, I also have to say, IReadOnlySessionState wont help you a whole lot in a high volume site because the SessionStateModule will still connect to SQL server to keep the session alive in ReleaseRequestState. So, best advice I would give is to use the new AppFabric Cache based session state or go stateless.

Best, Mike

DenisM 13 years ago |

In addition, when it comes to hanging on AWS EC2, remember to check your ICMP: http://www.daemonology.net/blog/2012-11-28-broken-EC2-firewa...

DenisM 13 years ago |

I used to have sporadic hangs when it didn't even look like the request reached the IIS server (hosted on EC2), but I didn't know where to start looking. Any advice on that?

mvolo 13 years ago | |

Hey DennisM,

The first thing I would check is that you can hit the webserver. You can do some of that with a tool like PortCheck: http://mvolo.com/check-iis-network-connectivity-with-portche....

Best, Mike

DenisM 13 years ago | | |

I'll try it next time the problem happens. Hope it works with SSL.

What I noticed is that when a hang appears, not even a record in the IIS log file is made until the hang clears up. All the while ping works just fine. I guess what I really want is to know when the IIS received the connection, but before running any ASP.NET code. That would be super helpful in narrowing down the problems.

ldubinets 13 years ago |

Whats this? Something related to .NET on HN front page? I must be dreaming...

mvolo 13 years ago | |

We share your sentiment! We are a large-scale startup built entirely on the MSFT stack, in Azure cloud, for the MSFT platform.

People on HN don't often talk about it, but it does happen, and apparently quite a bit judging by our users :)

ldubinets 13 years ago | | |

Yeah, the Microsoft stack plays well with startups (and enterprises too, obviously).

Everybody here seems to preach that "if you want to be in the startup scene you have GOT to get a MacBook and write Rails apps!" though.

graycat 13 years ago |

Okay, opinions of experts wanted!

So, apparently part of ASP.NET hangs is the use of SQL Server database to handle the data needed to support 'session state'.

Somehow, in my gut, for no very good, definite reason, I didn't want to involve SQL Server or any RDBMS in handling session state.

For a while I was going to handle session state with what is apparently the usual ASP.NET default, i.e., use 'application level' memory, that is, store session state in main memory, in an instance of a class which is a value in a collection class in memory common to the 'application', e.g., all ASP.NET Web pages running on that instance of Windows and/or IIS.

As I studied the ASP.NET support of session state, it seemed that I had to hurt my UI/UX by either (1) requiring that the user's browser accept cookies from my Web site or (2) have ASP.NET/IIS write some gibberish in a strange place in the URL of my Web site as my users see it. I didn't like either (1) or (2).

So, I thought of my solution: First, when first see the user, that is, when get their first HTTP request GET, and, thus, need a new 'session', get a new GUID value as a character string and use that as a 'session identifier' for that user.

Second, when send a page to the user, have that session ID value as the property TEXT in a field marked in HTML as 'hidden' so that it is not displayed by the user's browser.

Third, then I tried to use application memory to store session state 'my way': Declare my own class for session state. For each instance of session state, have an instance of that class. Store the instance as a value in a collection class used as a key-value store, with key the user's session ID, where the collection class is a value of the ASP.NET collection class in 'application memory'.

Then when get a postback from such a page, check the session ID. Now go get that user's 'session state'.

Somehow I couldn't get that use of application memory to work smoothly. There seemed to be a clumsy work-around.

So, instead of using application have a little console application that uses TCP/IP sockets and two instances of a collection class. Right: It's single threaded. Of course, one of the instances of the collection class is to be a key-value store where a key is a session ID and a value is a byte array of the serialized version of the instance for that user of my class for session state. For the other instance of the collection class, use it to store the time the session state was last touched (read or written) so that can know when can decree and declare that the session has 'expired'. Whenever the single thread is active, in addition to its main work for session state, it checks for expired sessions and deletes them.

So I did that. I wrote in VB.NET. Some code statistics are:

              file lines =    8412
             blank lines =    1739
           comment lines =    3524
            debug blocks =      56
         writeline lines =     251
              code lines =    3149
         code statements =    2108

It was fun code to write. I left the TCP/IP incoming queue length at the default of 100, but it could be raised.

I also implemented some 'system management' for the session state server: Any program able to use TCP/IP can send the session state server, on its IP port number, a request to report on number of sessions, to shut down, etc.

My timings indicate that on a dedicated 3.0 GHz single core processor, the code should be able to handle session state for sending 1000 Web pages a second, that is, write to the session state store and later read it back, this pair 1000 times a second.

Single threaded? SURE! Why not? Why interrupt the work on a single core processor?

On a multicore processor? Sure, start more than one executing instance of the little program and let each instance handle its own part of the full collection of possible GUID values used for session ID.

Yes, maybe I wrote a baby version of Redis. I may have written the code faster than I could have mastered Redis.

I've done some back of the envelope arithmetic that indicates that I will have one heck of a busy Web site before storing session state data in main memory would be a problem, especially given that even fast ECC main memory now goes for less than $9 per GB.

Where am I going wrong?

file lines = 8412 blank lines = 1739 comment lines = 3524 debug blocks = 56 writeline lines = 251 code lines = 3149 code statements = 2108