On the topic of plain text databases, I have been playing around with wordnet(basically the backbone data structure of a thesaurus ) and while there are a lot of perfectly good libraries to handle the data I was having fun building my own. One interesting thing is that the original data format has the byte offset of every record and link baked into the data, this makes it trivial to avoid having to load the whole thing into memory and you can directly seek to the record in question thus making it a plain text database. Admittedly one only good for reading as writes would have to rebuild all the indexes.
Honestly now days the whole thing can be trivially loaded into memory but back when the project was started this was much more of a concern, I do know that once I figured this out I started re writing my program to see how little memory I could use, It was a lot of fun to use access patterns other than "load the whole thing into memory"
Have a look at cdb. The more you read about its simplistic design the more you realize it is damn near the perfect solution for static and semi-static datasets. Fetches are either 1 or 2 disk reads depending on if the key exists.
I would love to be able to do `info recutils` before installing, as for deciding whether I want to install it. Is there someway to point info at some online source? (Yes I know there are published HTML versions of Texinfo documents, but I really want to do that in my terminal without needing to locate a website.)
I use the .rec format whenever I want a database maintained in git/github. The format is ideal if you want reasonable data diffs.
What tool do you use to read/write/modify? Do you do it manually?
I use VisiData[0] to view and modify in bulk. For simple error corrections I just edit the text file. For insertions as part of another process I'll write a simple script that appends a block of test to one of the .rec files.
(saulpw is the author of VisiData, and it's a marvelous piece of software.)
I guess I should have made that disclaimer myself, thanks for filling in the gap!
Thanks. Instantly installed as well. Free software is awesome (here: GNU, Linux, APT, and now recutils and visidata)!
Has support for recfiles improved recently? I was overjoyed when I learnt that VisiData had support for recfiles since I'd been getting tired of editing recfiles manually and liked VisiData's UI when working with CSVs and such.
But at least at the time (~2 years ago IIRC), the support was really basic, just basic record display, and most importantly, editing the parts VD didn't understand lead to data loss. I don't remember what I was trying to do - the error report I wrote with those details died with my old machine - but it wasn't anything too complicated, just array fields and foreign keys I believe i.e. just using recfile features one step beyond a listing of `key: value` pairs. I gave up on recfiles as a whole after losing data a few times like this (since I hadn't found any other suitable tool either).
Last I used Visidata, it didn't play nice with fields like %sort (they'd disappear if you re-saved the file) and if you had two fields with the same name in one record they'd get combined into a single field like "Name[2]:" when you re-saved. It might've also killed comments? I'm certainly not surprised it only has basic recfile support, because who use recfiles, but I'd be careful using VD with them for anything but viewing.
I haven't put the work into supporting full round-tripping, so yes, at the moment it's mostly useful for reading/viewing. If someone files an issue that would likely go a long way towards getting better support!
The front page "when you have .. but you need .." doesn't list rec files as a format. What other obscure formats does this support? I can't find a list
Amazingly rugged design that somehow places man and machine on equal footing by throwing us back to the time of casette futurism.
Amazingly poor choice of logo.
It’s turtles all the way down.
Those turtles are not highlighting a saying; they're having sex. They should be at a 0 degree angle on top of one another to represent the saying.
The turtle lives 'twixt plated decks
Which practically conceal its sex.
I think it clever of the turtle
In such a fix to be so fertile.
Yeah they go down so deep they start appearing from above.
I love recutils. The database format is simple enough, it has a bunch of options for constraints, and it has Bash integration and a great Emacs mode to search, edit and verify the integrity of the database.
Sure, it's not as fast as SQLite or bigger systems, but often it's enough for smaller projects.
Tortoise sex is a bold choice for a logo, but certainly memorable.
I like plaintext database files, and this looks way more readable than csv. But what I'd find even more interesting than improved readability would be features like atomicity. There are many tools for joins and other basic operations on tables stored in various text formats, but the real gap to 'proper' databases are ACID properties.
With recutils csv2rec I have my invoices list.
With recutils recsel | recfmt -f template.fodt I have my invoices.
soffice and curl to generate .pdf s and email them off.
With recutils recset I have my invoices status updated. Done.
For those who get blocked by gnu.org with a 403 (older Firefox) or an even sillier "Too Many Requests" error (older Safari) need to override their user agents strings to "curl" to make the site load again.
need to override their user agents strings to "curl" to make the site load again.
That seems very on-brand for them, as curl's default UA gets blocked by most sites.
In 2010 I remember people being very proficient with this at Amazon.
I really enjoying the toolset to query logs etc...
In the AI era, the rec file seems to be a great choice for formatting text that will be feed into LLMs. Imagine converting an HTML table into a RAG file, the context will be much clearer.
How avoiding LLM's altogether and using Emacs with Recutils for instance?
Thanks for the submission. I had no idea that existed, but I am definitely going to use this now.
As a similar, yet more powerful data format I started using Nickel (https://nickel-lang.org). It has very sophisticated typing and transformation features. Highly recommended!
rec files are not configuration, they are just a very simple and basic way of representing tabular data, very similar to a CSV. But it uses lines to separate columns and records, making it much simpler to add and edit records by hand, differently from CSV
recutils is aimed at database rather than config use-cases
2 turtles fucking is certainly a logo choice...
[flagged]
Leave Fred and George alone! They love each other.
I thought you was kidding first…
This among other things is why the GNU project as a whole has little credibility left.
Or "modern" people are too detached from reality.
Procreation of turtles (which this doesn't even show, there are just two turtles and turtles love to just walk on things including other turtles) is nothing secret.
Taste in logos—among the most dependable of measures for credibility!
As Mr Pratchett says - "it's turtles all the way down".
"Mommy, where do plaintext databases come from?"
"You see son, when a boy turtle and a girl turtle love each other very much..."
They were both boys if I recall correctly.
You recall correctly.
It's a bummer that the library and utilities are GPLv3 - really limits adoption, because it means that product developers can't build it into the kinds of small embedded Linux systems where it would really shine.
Have you tried reaching out to enquire if you could buy a version that was at least LGPL or proprietary licenced for you to bundle in your closed source application?
Or was this just a statement about being entitled to other peoples work and closing it up?
> product developers can't build it into the kinds of small embedded Linux systems
Are you sure about that? Because if you don't ship binaries, but whole devices, than only AGPL might demand what you think GPL does. Also I don't see what the issue is with "distributing" software from somebody else. If you designed things modular, the GPL software can be updated without the user needing to touch any of your proprietary code.
On the topic of plain text databases, I have been playing around with wordnet(basically the backbone data structure of a thesaurus ) and while there are a lot of perfectly good libraries to handle the data I was having fun building my own. One interesting thing is that the original data format has the byte offset of every record and link baked into the data, this makes it trivial to avoid having to load the whole thing into memory and you can directly seek to the record in question thus making it a plain text database. Admittedly one only good for reading as writes would have to rebuild all the indexes.
https://wordnet.princeton.edu/documentation/wndb5wn
Honestly now days the whole thing can be trivially loaded into memory but back when the project was started this was much more of a concern, I do know that once I figured this out I started re writing my program to see how little memory I could use, It was a lot of fun to use access patterns other than "load the whole thing into memory"
Have a look at cdb. The more you read about its simplistic design the more you realize it is damn near the perfect solution for static and semi-static datasets. Fetches are either 1 or 2 disk reads depending on if the key exists.
https://cdb.cr.yp.to/
https://news.ycombinator.com/item?id=22153665 505 points, 143 comments, 6 years ago
https://news.ycombinator.com/item?id=31832564 155 points, 52 comments, 3 years ago
https://news.ycombinator.com/item?id=15302035 105 points, 46 comments, 8 years ago
See also:
https://en.wikipedia.org/wiki/Recutils
I would love to be able to do `info recutils` before installing, as for deciding whether I want to install it. Is there someway to point info at some online source? (Yes I know there are published HTML versions of Texinfo documents, but I really want to do that in my terminal without needing to locate a website.)
If you're on Debian, you can do
Adapt the above to your system and its softwareI use the .rec format whenever I want a database maintained in git/github. The format is ideal if you want reasonable data diffs.
What tool do you use to read/write/modify? Do you do it manually?
I use VisiData[0] to view and modify in bulk. For simple error corrections I just edit the text file. For insertions as part of another process I'll write a simple script that appends a block of test to one of the .rec files.
[0] https://visidata.org
(saulpw is the author of VisiData, and it's a marvelous piece of software.)
I guess I should have made that disclaimer myself, thanks for filling in the gap!
Thanks. Instantly installed as well. Free software is awesome (here: GNU, Linux, APT, and now recutils and visidata)!
Has support for recfiles improved recently? I was overjoyed when I learnt that VisiData had support for recfiles since I'd been getting tired of editing recfiles manually and liked VisiData's UI when working with CSVs and such.
But at least at the time (~2 years ago IIRC), the support was really basic, just basic record display, and most importantly, editing the parts VD didn't understand lead to data loss. I don't remember what I was trying to do - the error report I wrote with those details died with my old machine - but it wasn't anything too complicated, just array fields and foreign keys I believe i.e. just using recfile features one step beyond a listing of `key: value` pairs. I gave up on recfiles as a whole after losing data a few times like this (since I hadn't found any other suitable tool either).
Last I used Visidata, it didn't play nice with fields like %sort (they'd disappear if you re-saved the file) and if you had two fields with the same name in one record they'd get combined into a single field like "Name[2]:" when you re-saved. It might've also killed comments? I'm certainly not surprised it only has basic recfile support, because who use recfiles, but I'd be careful using VD with them for anything but viewing.
I haven't put the work into supporting full round-tripping, so yes, at the moment it's mostly useful for reading/viewing. If someone files an issue that would likely go a long way towards getting better support!
The front page "when you have .. but you need .." doesn't list rec files as a format. What other obscure formats does this support? I can't find a list
https://visidata.org/formats
We keep that list up to date.
Amazingly rugged design that somehow places man and machine on equal footing by throwing us back to the time of casette futurism.
Amazingly poor choice of logo.
It’s turtles all the way down.
Those turtles are not highlighting a saying; they're having sex. They should be at a 0 degree angle on top of one another to represent the saying.
The turtle lives 'twixt plated decks
Which practically conceal its sex.
I think it clever of the turtle
In such a fix to be so fertile.
Yeah they go down so deep they start appearing from above.
I love recutils. The database format is simple enough, it has a bunch of options for constraints, and it has Bash integration and a great Emacs mode to search, edit and verify the integrity of the database.
Sure, it's not as fast as SQLite or bigger systems, but often it's enough for smaller projects.
Tortoise sex is a bold choice for a logo, but certainly memorable.
I like plaintext database files, and this looks way more readable than csv. But what I'd find even more interesting than improved readability would be features like atomicity. There are many tools for joins and other basic operations on tables stored in various text formats, but the real gap to 'proper' databases are ACID properties.
With recutils csv2rec I have my invoices list.
With recutils recsel | recfmt -f template.fodt I have my invoices.
soffice and curl to generate .pdf s and email them off.
With recutils recset I have my invoices status updated. Done.
For those who get blocked by gnu.org with a 403 (older Firefox) or an even sillier "Too Many Requests" error (older Safari) need to override their user agents strings to "curl" to make the site load again.
need to override their user agents strings to "curl" to make the site load again.
That seems very on-brand for them, as curl's default UA gets blocked by most sites.
In 2010 I remember people being very proficient with this at Amazon.
I really enjoying the toolset to query logs etc...
Good memories.
Reminds me of https://news.ycombinator.com/item?id=45458455
In the AI era, the rec file seems to be a great choice for formatting text that will be feed into LLMs. Imagine converting an HTML table into a RAG file, the context will be much clearer.
How avoiding LLM's altogether and using Emacs with Recutils for instance?
Thanks for the submission. I had no idea that existed, but I am definitely going to use this now.
As a similar, yet more powerful data format I started using Nickel (https://nickel-lang.org). It has very sophisticated typing and transformation features. Highly recommended!
rec files are not configuration, they are just a very simple and basic way of representing tabular data, very similar to a CSV. But it uses lines to separate columns and records, making it much simpler to add and edit records by hand, differently from CSV
recutils is aimed at database rather than config use-cases
2 turtles fucking is certainly a logo choice...
[flagged]
Leave Fred and George alone! They love each other.
I thought you was kidding first…
This among other things is why the GNU project as a whole has little credibility left.
Or "modern" people are too detached from reality.
Procreation of turtles (which this doesn't even show, there are just two turtles and turtles love to just walk on things including other turtles) is nothing secret.
Taste in logos—among the most dependable of measures for credibility!
As Mr Pratchett says - "it's turtles all the way down".
"Mommy, where do plaintext databases come from?"
"You see son, when a boy turtle and a girl turtle love each other very much..."
They were both boys if I recall correctly.
You recall correctly.
It's a bummer that the library and utilities are GPLv3 - really limits adoption, because it means that product developers can't build it into the kinds of small embedded Linux systems where it would really shine.
Have you tried reaching out to enquire if you could buy a version that was at least LGPL or proprietary licenced for you to bundle in your closed source application?
Or was this just a statement about being entitled to other peoples work and closing it up?
> product developers can't build it into the kinds of small embedded Linux systems
Are you sure about that? Because if you don't ship binaries, but whole devices, than only AGPL might demand what you think GPL does. Also I don't see what the issue is with "distributing" software from somebody else. If you designed things modular, the GPL software can be updated without the user needing to touch any of your proprietary code.
GPL doesn't extend to I/O.