Their content-addressing hash would seem critical, but the "combineUnordered" hash they use just adds each byte. The API is clear that this is only as good as possible, and I'm not sure I would rely on it for data used for security investigations. I suspect they'll come up with something like an arbitrary but fixed order over keys that would improve hash quality.
More generally, while I can maybe understand what they're doing, it's hard to imagine how to QA it in a way that's convincing to customers without a lot of data/compute/coverage analysis.
I suspect you're right and was already having similar thoughts regarding the hashing scheme. I put a patch together and am going to supplement with some additional tests of the collision space.
Unordered hashes made more sense for arbitrary Clojure data structures where the keys might be complex compound objects, but once we're in the land of datoms with finite value data types it's pretty easy to enforce a consistent ordering.
(disclaimer: I wrote much of the feature and post)
> where the keys might be complex compound objects
Given that any (simple) object has a unique c.a. identity, even anonymous (and unordered) containment in the parent object provides a key, that is k->k, with implicit order over {K} in the domain of keys (say ascending sort on k0..kn). As you obviously know, there are various schemas for that, such as "objspace://oid/<value-hash>". By definition, that is a bounded domain, with the same cardinality as the hash space. So then there remains the matter of nesting (aka trees) in which case unique identity of the parent is recursively computed as we walk up the tree from the leaves.
Re: hashing: Yes, but I'll leave that one to Paul who is a lot smarter than I am :)
Re: QA: can you say a bit more about the type of coverage you're worried about? Is your concern that we'd be missing APIs, or that the storage format itself breaks, resulting in fact elision? payne (the underlying project) has a borderline obnoxious amount of tests, but that doesn't mean we didn't miss anything :)
> in a way that's convincing to customers
Customers of this sort of security consulting are largely uninterested in the security as much as the audit report that allows them to say to their customers and incestors “we had these security professionals look at our stuff and this is what they said”.
Some of our customers, like Tailscale, are a helluva lot more picky than that.
Separate question: are there any new startups out there writing code primarily in Clojure?
Yes, in fact there is also a good number of conferences, for instance in less than two days https://2024.heartofclojure.eu/ starts in Belgium for instance (meet me there ;-) ).
We are writing a trading system for a small broker company in Clojure/ ClojureScript with a Datomic centered backend. The previous company some of us on the team worked at had the code-base also in Clojure/ ClojureScript.
(I'm one of the founders of Latacora and reviewed the post.) If any of you are at Heart of Clojure, I'll be there both wearing my Latacora hat and my Clojurists Together hat :)
How is the job market for juniors? Functional programming jobs seems to be anti-juniors when hiring.
Sadly Common Lisp, Scheme, Clojure, Julia, OCaml, F# and Haskell jobs are quite scarce.
For example, right now, I can only see ~50 LinkedIn EU Clojure ads. A dozen more mention Clojure but it does not seem to be the main focus of the job.
Would love to be proven wrong, though. Perhaps these jobs are sometimes not advertised via LinkedIn.
I would love for my next job to be in Common Lisp.
I do have a concern though. I see that at lots of companies the job is not a Go job or a Python job or a Javascript job, but a Go-at-Foocorp job or a Python-at-Bazco job or a Javascript-at-International-Clowns-Inc. job. What I mean is that the team have built so many leaky abstractions atop the core language, and added so many third-party tools, that the learning curve is steep and the skills non-transferable. On the one hand, who cares? We’re human, we can learn. But on the other it means that it can take significantly longer to ramp up and start delivering value than one would prefer.
What’s this have to do with Lisp? Well, while a well-written Lisp system will be faster to pick up and get started on that one in a less-powerful language (i.e., almost all of them), a poorly-written mess will be much, much slower. My concern is that I might leave a pretty good situation for what ends up being a dumpster fire — and then have to find a new job too quickly.
That's so true, you can write Fortran in any language! I've seen great Java codebases and atrocious Clojure out there.
Developing good non-over-engineered abstractions is hard. But with Lisp it's a bit easier.
Some places do abuse macro DSLs and you can end up, as you said, working on a local language with no transferable skills.
I'd say that market exists, but is kinda invisible.
I've been looking around for a junior with a passion for Common Lisp, but I have _no_ idea where to find one, especially in Germany (job's remote, but I'm currently not able to hire FTEs outside the country).
Hard to say, I was always contacted directly by the team/ had other role and later it turned out I can do a fair bit of programming too ;-)
That's like asking about the job market for hammers.
The annual Clojure/conj conference is coming up next month in Alexandria, VA (https://2024.clojure-conj.org) and there will be talks from a variety of companies, both big and small.
I was momentarily convinced that “crazy horse multiplier” must be an American idiom, and then realized that maybe the misspelled thing being multiplied is a force rather than a horse after all.
I usually catch these too and ninja edit them out
We (https://basil.net) are not new and not a startup, but our main language is Clojure and each new project is written in it. Very satisfied with our choice.
Yup, it's still the best tool for me. Especially for complex domains, as it does not allow you to over-abstract data structures.
Schemamap.io has a declarative SQL compiler and backend implemented in it, solving Postgres database syncing.
I use it for Wobaka[1]. It's 98% Clojure and ClojureScript with some bash added. I even use it for CSS with a beautiful library called Garden.
There are quite a few! We do at Doccla (and we’re hiring :) (EU))
havent touched clojure in years, but i use FP daily, can i get an interview? :)
I’ve sent you an email!
Anything for juniors?
Not at the moment, sorry!
> EU
:(
Definitely. It's a great tool, especially in a small and smart team.
Nalca Biotech
Fireproof is taking the lessons of cloujure and Ipfs and applying them to the browser, to create a realtime database that runs anywhere.
As cofounder of Couchbase I got to meet Rich Hickey, whose work was a big inspiration to Fireproof.
Content addressing makes causal consistency easy to implement. We are fast approaching readiness for consistency checks and ACID characterization.
It’s cool to see these ideas that you first conceptualized back at Couch, now becoming reality! I’m familiar with the high level, but not all the details. What lessons from Clojure and IPFS is Fireproof applying to the browser?
FYI: we're planning a followup post for people who are less interested in the Datomic mechanics and more interested in the usefulness to investigations and other security functions. Informally, I think of it this way: your SIEM has the deltas, but often you want the contextual states in between the deltas (and a lot of investigation is about trying to reason about that state). We built this tool originally to support that, and it turned out that approach was also super useful for things like compliance, CSPM...
Their content-addressing hash would seem critical, but the "combineUnordered" hash they use just adds each byte. The API is clear that this is only as good as possible, and I'm not sure I would rely on it for data used for security investigations. I suspect they'll come up with something like an arbitrary but fixed order over keys that would improve hash quality.
More generally, while I can maybe understand what they're doing, it's hard to imagine how to QA it in a way that's convincing to customers without a lot of data/compute/coverage analysis.
I suspect you're right and was already having similar thoughts regarding the hashing scheme. I put a patch together and am going to supplement with some additional tests of the collision space.
Unordered hashes made more sense for arbitrary Clojure data structures where the keys might be complex compound objects, but once we're in the land of datoms with finite value data types it's pretty easy to enforce a consistent ordering.
(disclaimer: I wrote much of the feature and post)
> where the keys might be complex compound objects
Given that any (simple) object has a unique c.a. identity, even anonymous (and unordered) containment in the parent object provides a key, that is k->k, with implicit order over {K} in the domain of keys (say ascending sort on k0..kn). As you obviously know, there are various schemas for that, such as "objspace://oid/<value-hash>". By definition, that is a bounded domain, with the same cardinality as the hash space. So then there remains the matter of nesting (aka trees) in which case unique identity of the parent is recursively computed as we walk up the tree from the leaves.
Re: hashing: Yes, but I'll leave that one to Paul who is a lot smarter than I am :)
Re: QA: can you say a bit more about the type of coverage you're worried about? Is your concern that we'd be missing APIs, or that the storage format itself breaks, resulting in fact elision? payne (the underlying project) has a borderline obnoxious amount of tests, but that doesn't mean we didn't miss anything :)
> in a way that's convincing to customers
Customers of this sort of security consulting are largely uninterested in the security as much as the audit report that allows them to say to their customers and incestors “we had these security professionals look at our stuff and this is what they said”.
Some of our customers, like Tailscale, are a helluva lot more picky than that.
Separate question: are there any new startups out there writing code primarily in Clojure?
Yes, in fact there is also a good number of conferences, for instance in less than two days https://2024.heartofclojure.eu/ starts in Belgium for instance (meet me there ;-) ).
We are writing a trading system for a small broker company in Clojure/ ClojureScript with a Datomic centered backend. The previous company some of us on the team worked at had the code-base also in Clojure/ ClojureScript.
(I'm one of the founders of Latacora and reviewed the post.) If any of you are at Heart of Clojure, I'll be there both wearing my Latacora hat and my Clojurists Together hat :)
How is the job market for juniors? Functional programming jobs seems to be anti-juniors when hiring.
Sadly Common Lisp, Scheme, Clojure, Julia, OCaml, F# and Haskell jobs are quite scarce.
For example, right now, I can only see ~50 LinkedIn EU Clojure ads. A dozen more mention Clojure but it does not seem to be the main focus of the job.
Would love to be proven wrong, though. Perhaps these jobs are sometimes not advertised via LinkedIn.
I would love for my next job to be in Common Lisp.
I do have a concern though. I see that at lots of companies the job is not a Go job or a Python job or a Javascript job, but a Go-at-Foocorp job or a Python-at-Bazco job or a Javascript-at-International-Clowns-Inc. job. What I mean is that the team have built so many leaky abstractions atop the core language, and added so many third-party tools, that the learning curve is steep and the skills non-transferable. On the one hand, who cares? We’re human, we can learn. But on the other it means that it can take significantly longer to ramp up and start delivering value than one would prefer.
What’s this have to do with Lisp? Well, while a well-written Lisp system will be faster to pick up and get started on that one in a less-powerful language (i.e., almost all of them), a poorly-written mess will be much, much slower. My concern is that I might leave a pretty good situation for what ends up being a dumpster fire — and then have to find a new job too quickly.
That's so true, you can write Fortran in any language! I've seen great Java codebases and atrocious Clojure out there.
Developing good non-over-engineered abstractions is hard. But with Lisp it's a bit easier.
Some places do abuse macro DSLs and you can end up, as you said, working on a local language with no transferable skills.
I'd say that market exists, but is kinda invisible.
I've been looking around for a junior with a passion for Common Lisp, but I have _no_ idea where to find one, especially in Germany (job's remote, but I'm currently not able to hire FTEs outside the country).
Hard to say, I was always contacted directly by the team/ had other role and later it turned out I can do a fair bit of programming too ;-)
That's like asking about the job market for hammers.
The annual Clojure/conj conference is coming up next month in Alexandria, VA (https://2024.clojure-conj.org) and there will be talks from a variety of companies, both big and small.
InstantDB (YC S22) launched a few weeks ago https://news.ycombinator.com/item?id=41322281
I don't know about startups, but I know of several more mature companies that are using it. It's, IMO, the best way to use the Java ecosystem.
Even if you never push the code to production, it’s a crazy gorse multiplier for the explore and adventure prototype phase of a project
"gorse multiplier" wins today's "best transcription error" award, IMO
category: edit-distance-two
2nd place probably goes to another comment in this thread: https://news.ycombinator.com/item?id=41560711
I was momentarily convinced that “crazy horse multiplier” must be an American idiom, and then realized that maybe the misspelled thing being multiplied is a force rather than a horse after all.
I usually catch these too and ninja edit them out
We (https://basil.net) are not new and not a startup, but our main language is Clojure and each new project is written in it. Very satisfied with our choice.
Yup, it's still the best tool for me. Especially for complex domains, as it does not allow you to over-abstract data structures.
Schemamap.io has a declarative SQL compiler and backend implemented in it, solving Postgres database syncing.
I use it for Wobaka[1]. It's 98% Clojure and ClojureScript with some bash added. I even use it for CSS with a beautiful library called Garden.
Extremely happy going with Clojure!
[1] https://wobaka.com/blog/building-a-startup-on-clojure/
There are quite a few! We do at Doccla (and we’re hiring :) (EU))
havent touched clojure in years, but i use FP daily, can i get an interview? :)
I’ve sent you an email!
Anything for juniors?
Not at the moment, sorry!
> EU
:(
Definitely. It's a great tool, especially in a small and smart team.
Nalca Biotech
Fireproof is taking the lessons of cloujure and Ipfs and applying them to the browser, to create a realtime database that runs anywhere.
As cofounder of Couchbase I got to meet Rich Hickey, whose work was a big inspiration to Fireproof.
Content addressing makes causal consistency easy to implement. We are fast approaching readiness for consistency checks and ACID characterization.
It’s cool to see these ideas that you first conceptualized back at Couch, now becoming reality! I’m familiar with the high level, but not all the details. What lessons from Clojure and IPFS is Fireproof applying to the browser?
FYI: we're planning a followup post for people who are less interested in the Datomic mechanics and more interested in the usefulness to investigations and other security functions. Informally, I think of it this way: your SIEM has the deltas, but often you want the contextual states in between the deltas (and a lot of investigation is about trying to reason about that state). We built this tool originally to support that, and it turned out that approach was also super useful for things like compliance, CSPM...
We already wrote a high level blog post here: https://www.latacora.com/blog/2023/11/01/our-approach-to-bui... -- but the one I'm hoping to write is more of a case study.