C++ vs JavaScript
it’s more like gpu vs CPU
Rockstar making GTA online be like: “Computer, here is a 512mb json file please download it from the server and then do nothing with it”
I have the same problem with XML too. Notepad++ has a plugin that can format a 50MB XML file in a few seconds. But my current client won’t allow plugins installed. So I have to use VS Code, which chokes on anything bigger than what I could do myself manually if I was determined.
I use vim macros. You can do some crazy formatting with it
You don’t need to open a file in a text editor to format it
Time to train an LLM to format XML and hope for the best
Do we need a “don’t parse xml with LLM” copypasta?
L arge L regex M odelI don’t wish death on anyone.
Wait, it’s all regex?
Always has been
Just install python and format it. Then
Meanwhile, I can open a 1GB file in (stock) vim without any trouble at all.
Formatting is what
xmllintis for.
I’ve never had any problems with 4,2 MB (and bigger) json files. What languages/libraries/editors chokes on it?
Notepad
Render the json as polygons?
That just results in an image of JSON Bourne.
It’s time someone wrote a JSON shader.
Ray TraSON
Rayson
Maybe it’s time we invent JPUs (json processing units) to equalize the playing field.
Until then, we have simdjson https://github.com/simdjson/simdjson
The best I can do is an ML model running on an NPU that parses JSON in subtly wrong and impossible to debug ways
Just make it a LJM (Large JSON Model) capable of predicting the next JSON token from the previous JSON tokens and you would have massive savings in file storagre and network traffic from not having to store and transmit full JSON documents all in exchange for an “acceptable” error rate.
Hardware accelerated JSON Markov chain operations when?
So you’re saying it’s already feature complete with most json libraries out there?
deleted by creator
Did you know? By indiscriminately removing every 3rd letter, you can ethically decrease input size by up to 33%!
JSON and the Argonaut RISC processors
Latest Nvidia co-processor can perform 60 million curly brace instructions a second.
60 million CLOPS? No way!
Finally, something to process “databases” that ditched excel for json!
That is sometime the issue when your code editor is a disguised web browser 😅
No, if you’re struggling to load 4.2 mb of text the issue is not electron.
Given it is a CPU is limiting the parsing of the file, I wonder how a GPU-based editor like Zed would handle it.
Been wanting to test out the editor ever since it was partially open sourced but I am too lazy to get around doing it
That’s not how this works, GPUs are fast because the kind of work they do is embarrassingly parallel and they have hundreds of cores. Loading a json file is not something that can be trivially parallelized. Also, zed use the gpu for rendering, not reading files.
I’d like to point out for those who aren’t in the weeds of silicon architecture, ‘embarrassingly parellel’ is the a type of computation work flow. It’s just named that because the solution was an embarrassingly easy one.
Huh, I was about to correct you on the use of embarrassment in that the intent was to mean a large amount, but it seems a Wiki edit reverted it to your meaning a year ago, thanks for making me check!
As far as my understanding goes, Zed uses the GPU only for rendering things on screen. And from what I’ve heard, most editors do that. I don’t understand why Zed uses that as a key marketing point
To appeal to people who don’t really understand how stuff works but think GPU is AI and fast
i hate to break it to you bud but all modern editors are GPU based
Fifty million polygons processed by over 7 thousand processing cores (Intel iGPU), versus 4 million tokens processed by a single execution unit (with some instruction reordering tricky).
Doesn’t a 3070 have less than 7k cores? A UHD 750 (relatively recent iGPU) only has 256.
And I don’t know the structure of JSON that well, but can’t tokens be made of multiple chars?
You’re right, I looked up the highest Intel GPU count but forgot that they released desktop cards. Intel iGPUs “only” have 768 cores, it’s the Ampere cards that have thousands of cores.
JSON is UTF-8 so it can be up to three bytes per token theoretically. Depends on the language you’re processing, I guess.
yea we need multithreaded json parsers
CUDA accelerated JSON parser is sorely needed
Someone just needs to make a GPU-accelerated JSON decoder
You jest, but i asked for a similar (but much simpler) vector / polygon model, and it generated it.
Would you rather have 100,000 kg of tasty supreme pizza, or 200 kg of steaming manure?
Choose wisely.
Careful, the 100,000 kg of pizza will turn into manure.
I figure I can probably convert about 10 kg into manure before it autoconverts into compost. Which is maybe even a worse problem.
200kg of steaming manure would be pretty sweet if you had a vegetable garden
Not sure I’d chose to use the word “sweet” here…
Not a day or two from harvest.
Not sure if I’m just missing a reference here, but if you choose the pizza you can have both.
Yeah but then that would be human manure which is a little (although not much) more dangerous to use on stuff you’re going to eat
The pizza can be used to feed some people but you really have to go fast and find hungry people
Manure can be sold easily
Everybody gangsta still we invent hardware accelerated JSON parsing
You mean like simdjson?
No. Verlilogjson.
https://ieeexplore.ieee.org/document/9912040 “Hardware Accelerator for JSON Parsing, Querying and Schema Validation” “we can parse and query JSON data at 106 Gbps”
I’m so impressed that this is a thing
Coming soon, JSPU
106 Gbps
They get to this result on 0.6 MB of data (paper, page 5)
They even say:
Moreover, there is no need to evaluate our design with datasets larger than the ones we have used; we achieve steady state performance with our datasets
This requires an explanation. I do see the need - if you promise 100Gbps you need to process at least a few Tbs.
Imagine you have a car powered by a nuclear reactor with enough fuel to last 100 years and a stable output of energy. Then you put it on a 5 mile road that is comprised of the same 250 small segments in various configurations, but you know for a fact that starts and ends at the same elevation. You also know that this car gains exactly as much performance going downhill as it loses going uphill.
You set the car driving and determine that, it takes 15 minutes to travel 5 miles. You reconfigure the road, same rules, and do it again. Same result, 15 minutes. You do this again and again and again and always get 15 minutes.
Do you need to test the car on a 20 mile road of the same configuration to know that it goes 20mph?
JSON is a text-based, uncompressed format. It has very strict rules and a limited number of data types and structures. Further, it cannot contain computational logic on it’s own. The contents can interpreted after being read to extract logic, but the JSON itself cannot change it’s own computational complexity. As such, it’s simple to express every possible form and complexity a JSON object can take within just 0.6 MB of data. And once they know they can process that file in however-the-fuck-many microseconds, they can extrapolate to Gbps from there
That’s why le mans exist, to show that 100m races with muscle cars are a farce
Based on your analogue they drive the car for 7.5 inches (614.4 Kb by 63360 inches by 20 divided by 103179878.4 Kb) and promise based on that that car travels 20mph which might be true, yes, but the scale disproportion is too considerable to not require tests. This is not maths, this is a real physical device - how would it would behave on larger real data remains to be seen.
There is acceleration for text processing in AVX iirc
Well, do you have dedicated JSON hardware?
There were XML DOM accelerators for a while. Might still be out there.
You don’t?
@Randelung @seaQueue well, i have dedicated JavaScript hardware (https://developer.arm.com/documentation/dui0801/h/A64-Floating-point-Instructions/FJCVTZS)
At this point ARM is a CISC architecture
No, that’s not what RISC is about. There was some early attempts to keep the number of instructions low–originally, ARM didn’t have a multiply instruction, and there’s still a bunch of microcontrollers you can buy that don’t have a divide instruction–but it was quickly abandoned as it’s just not that useful. It only holds back instructions that optimize common cases. Your compiler can implement multiplication by doing addition in a loop, but that’s not very efficient.
What really worked about it was keeping a separation between how memory is accessed. You don’t have an ADD instruction that can fetch from both registers or main memory. You have a MOV instruction that can fetch from memory into a register, and you have an ADD instruction that can work on registers.
ARM still does this just fine.
Someone confusing load-store with RISC again.
I’m a computer engineering major (still a student tbf), I’m well aware of the difference between CISC and RISC, I was making a joke.
Also, I understand your point, but you should know though that a load-store architecture and a RISC instruction set are not the same thing. The vast majority of RISC ISAs are load-store, but not all load-store architectures are RISC.
http://www.quadibloc.com/arch/sriscint.htm
The RISC architecture contains several common elements. Some of them are no longer present in most chips that still call themselves RISC:
- All instructions execute in a single cycle.
- Floating-point operations, specifically, are therefore excluded.
But most of the defining characteristics of RISC do remain in force:
- All instructions occupy the same amount of space in memory.
- Only load, store, and jump instructions directly address memory. Calculations are performed only between operands in registers.
https://groups.google.com/g/comp.arch/c/IZP5KUJprHw?pli=1
MOST RISCs:
3a) Have 1 size of instruction in an instruction stream
3b) And that size is 4 bytes
3c) Have a handful (1-4) addressing modes) (* it is VERY hard to count these things; will discuss later).
3d) Have NO indirect addressing in any form (i.e., where you need one memory access to get the address of another operand in memory)
4a) Have NO operations that combine load/store with arithmetic, i.e., like add from memory, or add to memory. (note: this means especially avoiding operations that use the value of a load as input to an ALU operation, especially when that operation can cause an exception. Loads/stores with address modification can often be OK as they don’t have some of the bad effects)
4b) Have no more than 1 memory-addressed operand per instruction
5a) Do NOT support arbitrary alignment of data for loads/stores
5b) Use an MMU for a data address no more than once per instruction
6a) Have >=5 bits per integer register specifier
6b) Have >= 4 bits per FP register specifierNote that none of this has to do with reducing the number of instructions, which is what people tend to think of when they hear the name.
All instructions occupy the same amount of space in memory.
Both ARM and RISC-V have compressed instructions. Dunno how ARM works but with RISC-V the 16-bit instruction set is freely interspersable with the 32 bit one, which also get their alignment reduced to 16 bits. Gets like 95% of the space reduction possible with full variable-width instructions without overcomplicating the insn decoder.
As to addressing and loads and arithmetic: No such instructions, but every CPU but the tiniest ones are expected to do macro-op fusion for things like indexed loads. Here’s an overview.
The MMU thing… well the vector extension can do gather/scatter, I guess it could stay within the letter of “use the MMU once” but definitely not the spirit.
The R in ARM and RISC is a lie.
The website title says “Arm Developer”, not “ARM Developer”, in a clearly non-acronym way so it’s a guide for making prosthetic hardware. Of course you want a cyborg arm to parse JS natively, why else even get one?
Lie starts with L, dummy
Nope it’s still a register-register op, that’s very much load-store architecture.
It’s reduced, not minimalist, otherwise every RISC CPU out there would only have one instruction like decrement and branch if nonzero. RISC-V would not have an extension mechanism. The instruction exists because it makes things faster because you don’t have to do manual bit-fiddling over 10 instructions to achieve a thing already-existing ALU logic can do in a single cycle. A thing that isn’t even javascript-specific (or terribly relevant to json), it’s a specific float to int cast with specific rounding and overflow mode. Would it more palatable to your tastes if the CPU were to do macro-op fusion on 10(!) instructions to get the same result?
Please no, don’t subsidize anything Java-Script. It will only make it less efficient.
And thus JsPU was born from Lemmy
NOOOOOOOO
Modern sites: This page requires a JsPU to run which is not present on this system. The website will run in reduced feature mode.
We did it!
I’ll take 10. Here is a picture of a goldfish as payment.
4 Js-core @ 2.00 GHz
Slap a liquid cooler and you’re cooking at a high speed 2.08 GHz
I love the idea that we could need a dedicated chip to parse web pages 😂
My thoughts on software in general over the past 20 years. So many programs inefficiently written and in 4th level languages just eats up any CPU/memory gain. (Less soap box and more of a curious what if to how fast things would be if we still wrote highly optimized programs)
Even if our apps used resources better the adware will just use the free space.
I fully concur. There’s tons of really inefficient software out there that wastes resources just because for a long time, available resources grew fast enough to just keep using more of them without the net speed of an application slowing down. If we didn’t have so many lazy SW devs, I suspect the reduction in needed CPU cycles would have a measurable positive effect on climate change.
Answer: there’d be far less software in the world, it would all be more archaic and less useful, and our phones and laptops would just sit at 2% utilization most of the time.
There’s an opportunity cost to everything, including fussing over whether that value can be stored as an int instead of a double to save 8 bits of space. High level languages let developers express their feature and business logic faster, with fewer bugs, and much lower ongoing maintenance costs.
Works fine in
vimExcept if it’s a single line file, only god can help you then. (Or running
prettier -won it before opening it or whatever.)cat file.json | jqalso worksRender Media works the best
rm file.jsonYes, Render Media is the best. It’s hard to believe that not many people know about this tool. It’s also natively installed in all Linux distros.
https://porkmail.org/era/unix/award#cat
jq < file.jsoncatis for concatenating multiple files, not redirecting single files.
4.2 megs on one line? Vim probably can handle it fine, although syntax won’t be highlighted past a certain point
I’ve accidentally opened enormous single line json files more than once. Could be lsp config or treesitter or any number of things but trying to do any operations after opening such a file is not a good time.
Technically every JSON file is a single line, with line break characters here and there
:syntax offand it works just fine.
