I wrote the code and the blog, happy to answer any questions/comments. Very eager to have folks try it out and give feedback! Like is my meme game strong or very strong? J/K
There's some missing bits around FFI and callbacks (i.e. C calling function pointer that is a luajit generated stub back into the interpreter) and curious if anyone actually uses these things in OpenResty workloads. Deploy and enjoy!
I'd love to use this to profile Neovim, where (nowadays) user scripts are often written in Lua 5.1. Neovim by default embeds LuaJIT (close to HEAD). Being able to see Lua functions and C functions in the stack would be amazing.
Is there a single-shot mode that I can use from the command-line? Something like:
$ parca-perf record nvim ...
Or:
a$ nvim ...
b$ parca-perf top -p $(pidof nvim)
There isn’t, because the indented way to use Parca is to profile production and always-on.
However, we wouldn’t be against adding a mode like this!
FWIW both the server and the agent are single statically linked binaries so while it’s a bit more set up it’s not terribly difficult either[1].
It should work to profile neovim, does it use the luajit shared library or is it statically linked? You could run parca-agent and set up a filter to discard all non-neovim samples and leave a parca instance running locally and just restart it periodically.
Thanks for submitting! We know HN has a sweet spot for LuaJIT, so we figured it would eventually end up here.
Quick summary: this post dives into the gory details of how we implemented an eBPF based profiler for LuaJIT.
Let us know if you have any questions on this, we’ll keep an eye out on comments!
[deleted]
Very deep dive, thank you for sharing it all. So cool it traverses callbacks too.
Glad you liked it! Yeah, we worked with a customer who really needs this badly and has done some unspeakable things to get by until now.
I’m tremendous excited about LuaJIT 3.0 development.
None of the items listed there have been ticked off in the time since the ticket was opened, not even the “create v3 branch” one. Mike also has had plans for v3 for at least the last decade too.
So, I’m sure it’ll get worked on when he can, and it’ll be great when it’s done, but it doesn’t look like there’s active development on it and it doesn’t look like it will happen any time soon. I hope in wrong, of course, but it just doesn’t seem likely.
From what I understood Mike does not want to merge someone else’s implementation of a new ISA but would rather be sponsored and do it himself. Can’t be bothered to source this claim at the moment so feel free to treat it as “came to me in a dream” level authenticity until proven otherwise. Seems reasonable though, I would also be paranoid about merging a sensitive complicated JIT implementation from an unknown contributor.
He wrote something along those lines here [1], which was in reply to a completely different, prototype-quality RISC-V port attempt
> Is the sponsor prepared to sponsor the initial review and integration into the LuaJIT default code base by me?
> Is the sponsor prepared to sponsor the inevitable initial bug fixes and the extra effort for continued maintenance that a new architecture entails?
Also, I should have been clearer about the new GC I linked to: I have not seen Mike say anything about it, and I wouldn't be surprised in the least if he rejects it and (wishes to) write his own, because he's had his own plans for many years. It seems impossible to get anything past him without modification. (I think it's a pity to see someone send a PR with a highly informative commit message and he replaces the body with "Thanks to X. #987")
I was a part of the team that contributed a few of the ports actually. For example, you can take a look at vm_mips64.dasc file header for the contributor list.
Though, it is possible that he changed his mind after having to review thousands of lines of assembly written by 25 year olds. :)
Kudos! Was it difficult to get it accepted? I've seen ports rejected.
To be honest, I forgot because it was quite some time ago. But I don't think we had any difficulties in that regard. I do remember being quite worried that it will not be good enough. In the time I started working on it, Mike sent a brutal email to a person trying to do PPC64 port (https://www.freelists.org/post/luajit/PPC64le-port-status,1).
LOL, vicious! I don’t feel sorry for them though — I learned a lot by getting a few harsh corrections when I was a young lad trying to run with bigger dogs.
LuaJIT.org stopped publishing release tarballs [1] which caused leafo's GH actions builds [2] to suddenly stop working. The workaround was to start testing against OpenResty's distribution of LuaJIT [3] which is incompatible with LuaJIT.org's version.
There is no faster way to make a fork the de facto standard version than to break everyone's CI builds.
The workaround for LuaJIT moving to Github was to... clone a fork of it?
If they could do that, why can't they just pull from the LuaJIT repo?
LuaJIT didn't move to GitHub, they just have a mirror there.
The thing is they stopped numbering and publishing releases, it's all a rolling release without any name or number, so you cannot snapshot I'm certain version.
But OpenResty fork does create tag versions with date, so they can build or test against certain concrete snapshot frozen in time.
> it's all a rolling release without any name or number, so you cannot snapshot I'm certain version
A git commit SHA is a number that identifies a version of the code.
Yep, but it doesn't serve as model of reference of date, order, patch or anything, and it's meaningless without the git database. It's OK as an immutable tag, but doesn't give you any additional information.
Even without SemVer, using a simple date versioning system (let's say 2024.10.2-patch4) gives you a lot of context.
At some point in my life (when I briefly worked on LuaJIT for DeepMind) I have written a stack walker which can stitch together native and Lua frames: for each native stack frame it checks if that is actually an interpreter frame or a trace frame - if that's the case it finds corresponding `lua_State` and unwinds corresponding Lua stack, then continues with native stack again.
This way you get a stack trace which contains all Lua and native frames. You can use it when profiling and you can use it to print hybrid stack traces when your binary crashes.
I was considering open-sourcing it, but it requires a bunch of patches in LJ internals so I gave up on that idea.
(There is also some amount of over-engineering involved, e.g. to compute unwinding information for interpreter code I run an abstract interpretation on its implementation and annotate interpreter code range with information on whether it is safe or unsafe to try unwinding at a specific pc inside the interpreter. I could have just done this by hand - but did not want to maintain it between LJ versions)
I wrote the code and the blog, happy to answer any questions/comments. Very eager to have folks try it out and give feedback! Like is my meme game strong or very strong? J/K
There's some missing bits around FFI and callbacks (i.e. C calling function pointer that is a luajit generated stub back into the interpreter) and curious if anyone actually uses these things in OpenResty workloads. Deploy and enjoy!
I'd love to use this to profile Neovim, where (nowadays) user scripts are often written in Lua 5.1. Neovim by default embeds LuaJIT (close to HEAD). Being able to see Lua functions and C functions in the stack would be amazing.
Is there a single-shot mode that I can use from the command-line? Something like:
Or:There isn’t, because the indented way to use Parca is to profile production and always-on.
However, we wouldn’t be against adding a mode like this!
FWIW both the server and the agent are single statically linked binaries so while it’s a bit more set up it’s not terribly difficult either[1].
[1] https://www.parca.dev/docs/quickstart/
It should work to profile neovim, does it use the luajit shared library or is it statically linked? You could run parca-agent and set up a filter to discard all non-neovim samples and leave a parca instance running locally and just restart it periodically.
Thanks for submitting! We know HN has a sweet spot for LuaJIT, so we figured it would eventually end up here.
Quick summary: this post dives into the gory details of how we implemented an eBPF based profiler for LuaJIT.
Let us know if you have any questions on this, we’ll keep an eye out on comments!
Very deep dive, thank you for sharing it all. So cool it traverses callbacks too.
Glad you liked it! Yeah, we worked with a customer who really needs this badly and has done some unspeakable things to get by until now.
I’m tremendous excited about LuaJIT 3.0 development.
https://github.com/LuaJIT/LuaJIT/issues/1092
Q: does anyone know timeline on the release?
Looks like 10 years to me
Why do you think that?
None of the items listed there have been ticked off in the time since the ticket was opened, not even the “create v3 branch” one. Mike also has had plans for v3 for at least the last decade too.
So, I’m sure it’ll get worked on when he can, and it’ll be great when it’s done, but it doesn’t look like there’s active development on it and it doesn’t look like it will happen any time soon. I hope in wrong, of course, but it just doesn’t seem likely.
Actually there is some progress.
For example there is a new higher-performance GC (https://github.com/LuaJIT/LuaJIT/issues/38#issuecomment-1696...) since a year ago (in fact, at least 3 people over the years have taken a stab at writing a new GC!)
And a full port to (certain flavours of) RISC-V was finished a couple months ago and awaiting merge (https://github.com/LuaJIT/LuaJIT/pull/1267), and might be merged separately into the OpenResty fork (https://github.com/openresty/luajit2/pull/236).
From what I understood Mike does not want to merge someone else’s implementation of a new ISA but would rather be sponsored and do it himself. Can’t be bothered to source this claim at the moment so feel free to treat it as “came to me in a dream” level authenticity until proven otherwise. Seems reasonable though, I would also be paranoid about merging a sensitive complicated JIT implementation from an unknown contributor.
He wrote something along those lines here [1], which was in reply to a completely different, prototype-quality RISC-V port attempt
> Is the sponsor prepared to sponsor the initial review and integration into the LuaJIT default code base by me?
> Is the sponsor prepared to sponsor the inevitable initial bug fixes and the extra effort for continued maintenance that a new architecture entails?
Also, I should have been clearer about the new GC I linked to: I have not seen Mike say anything about it, and I wouldn't be surprised in the least if he rejects it and (wishes to) write his own, because he's had his own plans for many years. It seems impossible to get anything past him without modification. (I think it's a pity to see someone send a PR with a highly informative commit message and he replaces the body with "Thanks to X. #987")
[1] https://github.com/LuaJIT/LuaJIT/issues/628#issuecomment-716...
I was a part of the team that contributed a few of the ports actually. For example, you can take a look at vm_mips64.dasc file header for the contributor list.
Though, it is possible that he changed his mind after having to review thousands of lines of assembly written by 25 year olds. :)
Kudos! Was it difficult to get it accepted? I've seen ports rejected.
To be honest, I forgot because it was quite some time ago. But I don't think we had any difficulties in that regard. I do remember being quite worried that it will not be good enough. In the time I started working on it, Mike sent a brutal email to a person trying to do PPC64 port (https://www.freelists.org/post/luajit/PPC64le-port-status,1).
LOL, vicious! I don’t feel sorry for them though — I learned a lot by getting a few harsh corrections when I was a young lad trying to run with bigger dogs.
LuaJIT.org stopped publishing release tarballs [1] which caused leafo's GH actions builds [2] to suddenly stop working. The workaround was to start testing against OpenResty's distribution of LuaJIT [3] which is incompatible with LuaJIT.org's version.
There is no faster way to make a fork the de facto standard version than to break everyone's CI builds.
[1] https://luajit.org/download.html
[2] https://github.com/leafo/gh-actions-lua/issues/49
[3] https://github.com/openresty/luajit2
The workaround for LuaJIT moving to Github was to... clone a fork of it?
If they could do that, why can't they just pull from the LuaJIT repo?
LuaJIT didn't move to GitHub, they just have a mirror there.
The thing is they stopped numbering and publishing releases, it's all a rolling release without any name or number, so you cannot snapshot I'm certain version.
But OpenResty fork does create tag versions with date, so they can build or test against certain concrete snapshot frozen in time.
> it's all a rolling release without any name or number, so you cannot snapshot I'm certain version
A git commit SHA is a number that identifies a version of the code.
Yep, but it doesn't serve as model of reference of date, order, patch or anything, and it's meaningless without the git database. It's OK as an immutable tag, but doesn't give you any additional information.
Even without SemVer, using a simple date versioning system (let's say 2024.10.2-patch4) gives you a lot of context.
At some point in my life (when I briefly worked on LuaJIT for DeepMind) I have written a stack walker which can stitch together native and Lua frames: for each native stack frame it checks if that is actually an interpreter frame or a trace frame - if that's the case it finds corresponding `lua_State` and unwinds corresponding Lua stack, then continues with native stack again.
This way you get a stack trace which contains all Lua and native frames. You can use it when profiling and you can use it to print hybrid stack traces when your binary crashes.
I was considering open-sourcing it, but it requires a bunch of patches in LJ internals so I gave up on that idea.
(There is also some amount of over-engineering involved, e.g. to compute unwinding information for interpreter code I run an abstract interpretation on its implementation and annotate interpreter code range with information on whether it is safe or unsafe to try unwinding at a specific pc inside the interpreter. I could have just done this by hand - but did not want to maintain it between LJ versions)
Lua and Nginx are fantastic. Did you know it's possible to add behavior/code lua for openrest dynamically? https://github.com/leandromoreira/lua-resty-dynacode?tab=rea...