The author seems to be have written this post based on work for an in-browser video editor (https://videoinu.com/editor). It runs completely on the client-side, so all the editing, previewing, rendering, and exporting is done using web technologies. Of course, it's not comparable to a professional editing tools but nevertheless that's very impressive achievement in my opinion!
It's true, a proper desktop editing application would probably be better in most cases, but I guess it would still be useful for the case where your computer can't run desktop applications (ex. chromebooks) or your administrator doesn't give you permission to install such applications.
> Because you'd have to download an app like this and then trust that it won't install malware or spyware or do anything else malicious.
How you would check that web app, which JavaScript/WebAssembly source/binary should be firstly loaded into your browser from 3rd-party website, would not "install malware or spyware or do anything else malicious"?
Modern browsers are pretty secure nowadays. By comparison, you have pretty much zero guarantees about an app you install onto your computer. The only concern is that these browser apps are quietly sending your files elsewhere, but this is easily inspectable via the dev tools.
Or you just run the installer in a sandbox/container, as has been possible for at least a decade in all major OSs
Sandboxee was released in 2004, and there have been various bits of registry journaing/snapshot software since the late 1990's which combined with NT ACLs could provide at least as much security as your getting out of a browser sandbox.
Browsers have an absolutely huge sandbox surface and lack even the most basic of fine-grained security controls out of the box. Enabling even basic restrictions to avoid fingerprinting has turned out to be difficult. How you do you know (particularly with firefox) that one tab isn't actively picking out data from another tab open to your banking site. Much less the problem with malicious javascript being injected via the ad frameworks.
> Or you just run the installer in a sandbox/container, as has been possible for at least a decade in all major OSs
The level of technical know-how necessary to accomplish that is astronomical compared to "here's a link," especially since it's cross-platform out of the box. That's why it's a no-brainer: any sane person is going to prefer that over installing untrusted software. Basically zero users know about the sandboxing solutions available for their system, much less willing put in the effort to Do It Right.
Has you ever read changelogs of Firefox & Chrome/Chromium? There are tons of issues documented related to sandboxing & security each new release happen.
Why would you use browser-based video editing over desktop-based free software, where you can read the source (or listen to someone you trust who has read the source) to make sure it's not doing anything malicious? Besides the fact that it might be easier/simpler for very small projects.
> (or listen to someone you trust who has read the source)
By this I meant to include, for example, the Debian team. Of course there can be bugs/oversights, but the same is true of the browser solution. Is there even one instance of malware being distributed through the official channels of a mainstream Linux distro? Not that an in browser JS video editor isn't cool, but I don't see how it would have technical advantages over the various Linux desktop editors.
So many of the projects I see mentioned here are neat hacks, but that's really about it.
People writing a complete operating system for an 8 bit micro made of discrete chips? That's neat, but not really useful, and most people won't ever take the time to boot it once.
An FPGA that runs web assembly code? That's nifty, but if we want high performance we probably won't use web assembly to do it.
You invented a completely new language that has all the features of C but none of the things that are annoying? We'll add it to the list of C-like-but-not-annoying languages.
Found a great new way to estimate how much capital you should try to get when creating a start-up to sell your all-natural soap products in California? That's great for you... a little bit niche for the rest of us.
You re-implemented emacs in python? That's a really impressive technical achievement. No one will ever surpass you in either emacs implementation or python hacks. Also, no one will ever use it.
Same thing with Hackaday... lots and lots of "cool" projects where neither the creator nor the site mods seem to realize something is just a nifty project, and not the salvation of the world.
You write as if these are bad things, but they're wonderful things. They're also the bread and butter of this site, so it's good that you see them here. Something would be wrong if you didn't.
Pure interest and sublime pointlessness are, of course, two of the classic hacker motivations.
I see such "not really useful" projects much as I do fine art. There's not much measurable use to the world in most painting, sculpture, or music in and of itself - but done well, it brings enjoyment to both the creator and appreciative observers. Count me as an appreciative observer in many such cases.
This would be accessible anywhere on any computer hooked up to the internet, even potentially a phone, tablet, raspberry pi, your grandma's computer, etc.
Many programs that don't need low level access like spreadsheets, pdf readers, word processors, image editors, etc.could be made into single page html files that would both be sandboxed and cross platform.
Software Engineer and Developer with a lot of experience in the area of full stack web development. I have been designing and building scalable backend servers and web frontends for more than 6 years.
> It's for building specialized services such as key value stores, L7 proxies, static websites, etc.
First of all, thank you for publishing this project. It's very interesting in my opinion since I never thought about the benefits of an event loop. Would you mind explaining briefly why an event loop is a better suit for these applications? Is it due to performance and efficiency?
I'd suggest that's not the right way to look at it. To a first approximation, "everything" is using an event loop nowadays, in that everything is using the same fundamental primitives to handle and dispatch events. In particular, this includes the Go runtime; run "strace" on a Go network program and you'll see these same calls pop up in the strace.
What this does instead is give a Go program direct access to the event loop. The benefit is that it bypasses all of the stuff that Go wraps around the internal event loop call that allows it to implement the way it offers a thread-like interface for you, and integrates with the channel and concurrency primitives, and maintains your position in the call stack between events, etc. The penalty is... the exact same thing, that you lose all the nice stuff that the Go runtime offers to you to implement the thread-like interface, etc., and are back to a lower-level interface that offers less services.
The performance of the Go runtime is "pretty good", especially by scripting language standards, but if you have sufficiently high performance requirements, you will not want to pay the overhead. The pathological case for all of these nice high-level abstractions is a server that handles a ton of network traffic of some sort and needs to do a little something to every request, maybe just a couple dozen cycle's worth of something, at which point paying what could be a few hundred cycles for all this runtime nice stuff that you're not using becomes a significant performance drain. Most people are not doing things where they can service a network request in a few dozen cycles, and the longer it takes to service a single request the more sense it makes to have a nice runtime layer providing you useful services, as it drops in the percentage of CPU time consumed by your program. For the most part, if you are so much as hitting a database over a network connection, even a local one, in your request, you've already greatly exceeded the amount of time you're paying to the runtime, for instance.
It does seem to me that a lot of people are a bit bedazzled by the top-level stuff that various languages offer, and forget that under the hood, everyone's using the event-based interfaces. What differs between Node and Twisted and all of the dozens or hundreds of other viable wrappers over these calls is the services automatically provided, not whether or not they are "event loops". Go is an event loop at the kernel level. Node is an event loop at the kernel level. Erlang is an event loop at the kernel level. They aren't all the same, but "event-based" vs. "not event-based" is not the distinction; it's a question of what they lay on top of the underlying event loop, not whether they use it. Even pure OS threads are, ultimately, event loops under the hood, just in the kernel rather than the user space.
> It does seem to me that a lot of people are a bit bedazzled by the top-level stuff that various languages offer, and forget that under the hood, everyone's using the event-based interfaces.
Yup. It's all very similar under the hood.
The most important difference between I/O models is whether the paradigm involves explicit vs. implicit management of the event loop. Callback models like Node, async/await style models like those of C#, and low-level primitives like IOCP, epoll, and kqueue fall into the former category. Go/Erlang, plain old threads, and even Unix processes fall into the latter category. There are advantages and disadvantages of each model.
Within each of these broad categories, the distinctions are, IMHO, much less interesting, and they're often made out to be more significant than they actually are. In particular, the distinction between runtimes like Go and regular OS pthreads is often made out to be more important than it really is, when the difference ultimately boils down to the CPU privilege level that thread management runs at.
Patrick, on the 2.6+ Linux kernels, is there a significant difference between threads and processes? It seems like both threads and processes are created via clone and the only difference is memory access?
I often hear "context switching between threads is cheaper" but pthreads still have their own PID and everything, so is this really the case?
Is there really much advantage to pthreads over the way PostgreSQL does things with efficient CoW sharing between processes for the binary?
The significance of the distinction depends entirely on the use case.
Yes, they’re both created with clone, but with different levels of sharing. A pthread will share the virtual address space of its parent, which makes shared memory simple to implement; use the same pointer and you’re done. CoW is not “sharing” really, because you can’t communicate over it, it just saves some creation overhead.
With CoW, technically nothing gets copied initially, but as soon as the new process starts executing, it’s going to start copying the stack frame and any other regions it’s using. With a pthread you can be certain it will just copy the stack.
Context switches are usually cheaper when you don’t need to throw out the old virtual address space (and invalidate the Translation Lookaside Buffer). Pthreads share virtual address space, so there is no need to flush the TLB.
In a use case like Postgres, you don’t necessarily need to optimise for context switches. If you have a lot of concurrent connections, each of which has one process, then you’ll only hit limits with context switching overhead if very few of those connections are fighting over any locks or spending much time in IO at all. This is atypical, so usually those other factors hit you first.
> The significance of the distinction depends entirely on the use case.
Indeed.
> Context switches are usually cheaper when you don’t need to throw out the old virtual address space (and invalidate the Translation Lookaside Buffer). Pthreads share virtual address space, so there is no need to flush the TLB.
I believe the cost of that has been reduced somewhat due to tagged TLBs on modern hardware.
> In a use case like Postgres, you don’t necessarily need to optimise for context switches. If you have a lot of concurrent connections, each of which has one process, then you’ll only hit limits with context switching overhead if very few of those connections are fighting over any locks or spending much time in IO at all. This is atypical, so usually those other factors hit you first.
Yea. There's a number of limitations in postgres due to the process model, but they're imo not TLB / context switch related. The biggest issue is that dynamically sharing memory between processes is harder, because there's no guarantee that it's possible for all post-fork memory allocations can portably be put at the same virtual addresses. Which then makes it more complicated to have shared datastructures, because you need to use relative pointers and such. That's not a problem for the main buffer pool etc, which is allocated when postgres is started, but it is problematic e.g. for memory shared between multiple processes working on the same query (say the memory for a shared hashtable in a hashjoin).
> I don't think this qualifies as a performance overhead, though, beyond the odd isub.
It ends up as one. The reason is less the additional instruction(s), but that you actually need to ferry arround additional data. In common scenarios you'll end up with a number of mappings shared between processes, so you can't just assume a single base address per-process. Instead you've to associate the specific mapping with relative pointers, and that does add to overhead. Both programming wise and runtime efficiency wise.
> I've done lots of event loops in the past (eg hellepoll in c++) and think that the cost of that is on the programmer - keeping track of things, callbacks, state machines and things and avoiding using the stack for state etc is all hard work and easy to mess up.
I very much agree. In the past, I have had quite some fun developing a few streaming parsers using Node.js, which also uses an event loop. And while these parser worked relatively good and efficient, debugging them was not an easy task. In addition, understanding the code is also a though challenge, especially for people other than the original authors.
When I started using Go more and more, I really enjoyed the different I/O-model using goroutines and blocking function calls. It also has a few drawbacks but the mental model is a lot easier to reason about.
People, I followed on Twitter, kept sharing links mentioning HN or citing from comments. After some time I decided to check it about but it took me some time until I realized what this "thing" was and appreciate the value behind it.
At my workplace, we are currently looking for a distributed locking mechanism. Since we are already heavily using Redis in our system, it seems to be convenient to rely on it, too, but we heart a lot of criticism against it.
Does anyone have experience with distributed locks and would be kind enough to share?
In a real-time system at work, we originally used Redis as a locking mechanism to avoid resource contention. However, it turned out the better long-term solution was to better use our reliable message broker and make tasks finer grain.
So, rather than lock large resources, we would break the processing into tiny tasks, each completely contained in a message object. Each message specified a new message to launch on completion. This way the messaging protocol is preventing resource contention rather than a lock.
[edit] The context involved some licenses software to which we only owned several instances. The original approach would attempt to grab a lock and run the batch. The final approach just created a queue for jobs and one consumer for each available license. Now, none of our compute is spent waiting on locks either.
Splitting a resource into multiple parts and embedding in into a single message in order to avoid external storage and the need for locking is an interesting approach, but sadly not an option for me. Nevertheless, I appreciate your response.
If the data size is limiting though, you can embed a Redis key or Hadoop FS reference into the message, rather than breaking your message system with massive data.
Then, you can have the storage system listen for completion messages, and cleanup appropriately. Or, like we did, just shove out old data LRU style. This way some history is always available for debugging.
Funnily, there is no PHP 6. While work had begun for this version, it was later abandoned due to some major issues regarding Unicode handling, as far as I know. Most of the features already implemented in PHP 6 were shipped in PHP 5.4 and they continued with PHP 7 in order to avoid confusion.
Indeed, resumable uploads are not a thing we invented. Nevertheless, we think it's handy to have a document describing the approach and client and server implementations.
We also considered the use of the Content-Range headers for resuming uploads but dropped this idea in favour of custom headers since the purpose written down in the HTTP specification was not compatible with the way we wanted to use it.
Great to hear, you enjoy it. I am Marius, the current project lead of the tus project and I guess I can provide some answers for you.
The biggest benefit and the main reason of its origin is the ability to resume an interrupted upload at any time resulting in more efficient resource usage and user experience.
In addition the fact that it's developed openly and by a community is also noticeable. You do not have to reinvent the wheel since others have already spend time, efforts and thoughts building a working system you can use for free. The tus project already offers a growing list of client and server-side implementations.
Last but not least, optional features such as parallel uploading and checksums are available.
On the other hand, you have the overhead of adapting an external system. Whenever you want to build software, you can choose between doing it yourself or using an existing solution, where latter can be tough to achieve depending on your architecture. Furthermore, tus introduces a small overhead in terms of numbers of requests needed for an upload. Usually, you need one to create the upload and another one to transfer the actual data (excluding eventual CORS requests in a browser), in contrast to the traditional model of uploading where a single request is used. Normally, this overhead is negligible, especially when the files are getting bigger.
No, I do not think this should be implemented. The proper way (in addition to server-side validation, of course) is to utilize JavaScript in order to verify the file's size. This solution also allows different maximum sizes for different file types as one example in the discussion shows. Using JavaScript also adds additional flexibility when trying to achieve custom error messages.
In general, I think the HTML specification is getting quite bloated these days, but this is not the place to discuss this.
