Hi!
We launched a new site last week that uses ClearScript to server render a React application. After launch, we get around 2-3 crashes on Access Violation Exception in w3wp.exe a day per web server. The site has around 200 concurrent users on average, with peaks up over a thousand. We use the latest version of ClearScript (5.4.7).
We see that the crash is an Access Violation Exception in v8-x64.dll and we have a few dump files from such crashes, but since we don't have any debugging symbols the stack traces doesn't give us much.
Would you be interested in looking at the crash dumps to help us figure this out? Or if you can help us with debug symbols so we can get more information out of the dumps.
Some information about how we use ClearScript:
1. We create multiple V8Runtime and V8ScriptEngine and keep them in a queue. When a request comes in, we dequeue an engine, run JS and return it back to the queue.
2. Since we don't store any JS globals we reuse the same V8 context for different visitors. But we run CollectGarbage(true) for every 30 execution. We have deployed a change this morning where we call CollectGarbage(false) instead but twice as often. Not because we think it will solve it, but see if the behavior changes.
3. We don't pass any CLR references to V8. The only thing we do is call Engine.Execute with strings of JS code. When we pass data to V8 we do that by either calling Engine.Execute or Engine.Evaluate with strings of JSON.
Any help or pointers are appreciated! We'd be happy to share code with you if you need it.
Comments: Hi again, >At first we had ~20 engines and runtimes that we never disposed, we just called CollectGarbage(true) on them for every 30 execution. This gave a slow buildup of memory. About one GB an hour. This gave us Access Violation crashes every 2-6 hours. We'd still like to see a stack for that, preferably with symbols, although it _sounds_ as if a runtime is simply running out of memory. Even the 64-bit V8 limits runtimes to about 1.5 GB by default, and in our experience there's only so far you can push it by adjusting the constraints, as V8 appears to have a number of fixed-size internal buffers that aren't under the host's control. It also reserves huge blocks of address space for each runtime, although that shouldn't be a problem in 64-bit. In any case, V8 seems to treat all resource exhaustion conditions as fatal and potentially process-corrupting, so its response is always to kill the process immediately via an unrecoverable access violation. >We only use ScriptEngine.Execute and ScriptEngine.Evaluate when we pass in JSON. For the main chunk of code we precompile at startup. Can you elaborate on that? What calls do you make to precompile the code? Subsequently, how do you invoke the precompiled code? And how do you pass JSON to it? A couple of small samples might help identify potential optimizations. >As I'm reading my own comments I see that I might come off as a bit aggressive which is very much not my intention. Not at all! We're very interested in making ClearScript useful on the server, and we appreciate your help! Cheers!
We launched a new site last week that uses ClearScript to server render a React application. After launch, we get around 2-3 crashes on Access Violation Exception in w3wp.exe a day per web server. The site has around 200 concurrent users on average, with peaks up over a thousand. We use the latest version of ClearScript (5.4.7).
We see that the crash is an Access Violation Exception in v8-x64.dll and we have a few dump files from such crashes, but since we don't have any debugging symbols the stack traces doesn't give us much.
Would you be interested in looking at the crash dumps to help us figure this out? Or if you can help us with debug symbols so we can get more information out of the dumps.
Some information about how we use ClearScript:
1. We create multiple V8Runtime and V8ScriptEngine and keep them in a queue. When a request comes in, we dequeue an engine, run JS and return it back to the queue.
2. Since we don't store any JS globals we reuse the same V8 context for different visitors. But we run CollectGarbage(true) for every 30 execution. We have deployed a change this morning where we call CollectGarbage(false) instead but twice as often. Not because we think it will solve it, but see if the behavior changes.
3. We don't pass any CLR references to V8. The only thing we do is call Engine.Execute with strings of JS code. When we pass data to V8 we do that by either calling Engine.Execute or Engine.Evaluate with strings of JSON.
Any help or pointers are appreciated! We'd be happy to share code with you if you need it.
Comments: Hi again, >At first we had ~20 engines and runtimes that we never disposed, we just called CollectGarbage(true) on them for every 30 execution. This gave a slow buildup of memory. About one GB an hour. This gave us Access Violation crashes every 2-6 hours. We'd still like to see a stack for that, preferably with symbols, although it _sounds_ as if a runtime is simply running out of memory. Even the 64-bit V8 limits runtimes to about 1.5 GB by default, and in our experience there's only so far you can push it by adjusting the constraints, as V8 appears to have a number of fixed-size internal buffers that aren't under the host's control. It also reserves huge blocks of address space for each runtime, although that shouldn't be a problem in 64-bit. In any case, V8 seems to treat all resource exhaustion conditions as fatal and potentially process-corrupting, so its response is always to kill the process immediately via an unrecoverable access violation. >We only use ScriptEngine.Execute and ScriptEngine.Evaluate when we pass in JSON. For the main chunk of code we precompile at startup. Can you elaborate on that? What calls do you make to precompile the code? Subsequently, how do you invoke the precompiled code? And how do you pass JSON to it? A couple of small samples might help identify potential optimizations. >As I'm reading my own comments I see that I might come off as a bit aggressive which is very much not my intention. Not at all! We're very interested in making ClearScript useful on the server, and we appreciate your help! Cheers!