Hi!
We launched a new site last week that uses ClearScript to server render a React application. After launch, we get around 2-3 crashes on Access Violation Exception in w3wp.exe a day per web server. The site has around 200 concurrent users on average, with peaks up over a thousand. We use the latest version of ClearScript (5.4.7).
We see that the crash is an Access Violation Exception in v8-x64.dll and we have a few dump files from such crashes, but since we don't have any debugging symbols the stack traces doesn't give us much.
Would you be interested in looking at the crash dumps to help us figure this out? Or if you can help us with debug symbols so we can get more information out of the dumps.
Some information about how we use ClearScript:
1. We create multiple V8Runtime and V8ScriptEngine and keep them in a queue. When a request comes in, we dequeue an engine, run JS and return it back to the queue.
2. Since we don't store any JS globals we reuse the same V8 context for different visitors. But we run CollectGarbage(true) for every 30 execution. We have deployed a change this morning where we call CollectGarbage(false) instead but twice as often. Not because we think it will solve it, but see if the behavior changes.
3. We don't pass any CLR references to V8. The only thing we do is call Engine.Execute with strings of JS code. When we pass data to V8 we do that by either calling Engine.Execute or Engine.Evaluate with strings of JSON.
Any help or pointers are appreciated! We'd be happy to share code with you if you need it.
Comments: Thanks for replying! After reading my own comment I see that I was a bit unclear. At first we had ~20 engines and runtimes that we never disposed, we just called CollectGarbage(true) on them for every 30 execution. This gave a slow buildup of memory. About one GB an hour. This gave us Access Violation crashes every 2-6 hours. Then we changed to run CollectGarbage(false) but for every 15 execution. This seemed to reduce the number of crashes but it made the memory usage increase a lot and we had to manually restart one server as the memory grew to 100%. After that we changed to dispose both the engine and the runtime after 300 executions. This has removed the crashes entirely, and the servers now use ~30% memory without it growing. We only use ScriptEngine.Execute and ScriptEngine.Evaluate when we pass in JSON. For the main chunk of code we precompile at startup. Our code does not leave artifacts behind that grows and it does not add global variables. I wouldn't call it pure (as in no side effects) but the process is "pass in a big JSON structure, get HTML back". The big JSON object passed in is stored in a local variable which no parts of the code references when the HTML is returned.
We launched a new site last week that uses ClearScript to server render a React application. After launch, we get around 2-3 crashes on Access Violation Exception in w3wp.exe a day per web server. The site has around 200 concurrent users on average, with peaks up over a thousand. We use the latest version of ClearScript (5.4.7).
We see that the crash is an Access Violation Exception in v8-x64.dll and we have a few dump files from such crashes, but since we don't have any debugging symbols the stack traces doesn't give us much.
Would you be interested in looking at the crash dumps to help us figure this out? Or if you can help us with debug symbols so we can get more information out of the dumps.
Some information about how we use ClearScript:
1. We create multiple V8Runtime and V8ScriptEngine and keep them in a queue. When a request comes in, we dequeue an engine, run JS and return it back to the queue.
2. Since we don't store any JS globals we reuse the same V8 context for different visitors. But we run CollectGarbage(true) for every 30 execution. We have deployed a change this morning where we call CollectGarbage(false) instead but twice as often. Not because we think it will solve it, but see if the behavior changes.
3. We don't pass any CLR references to V8. The only thing we do is call Engine.Execute with strings of JS code. When we pass data to V8 we do that by either calling Engine.Execute or Engine.Evaluate with strings of JSON.
Any help or pointers are appreciated! We'd be happy to share code with you if you need it.
Comments: Thanks for replying! After reading my own comment I see that I was a bit unclear. At first we had ~20 engines and runtimes that we never disposed, we just called CollectGarbage(true) on them for every 30 execution. This gave a slow buildup of memory. About one GB an hour. This gave us Access Violation crashes every 2-6 hours. Then we changed to run CollectGarbage(false) but for every 15 execution. This seemed to reduce the number of crashes but it made the memory usage increase a lot and we had to manually restart one server as the memory grew to 100%. After that we changed to dispose both the engine and the runtime after 300 executions. This has removed the crashes entirely, and the servers now use ~30% memory without it growing. We only use ScriptEngine.Execute and ScriptEngine.Evaluate when we pass in JSON. For the main chunk of code we precompile at startup. Our code does not leave artifacts behind that grows and it does not add global variables. I wouldn't call it pure (as in no side effects) but the process is "pass in a big JSON structure, get HTML back". The big JSON object passed in is stored in a local variable which no parts of the code references when the HTML is returned.