Same result would be achieved by paying tons of $$ for a high-end machine with dozens of cores, just to scrape one URL per core.
They could have implemented asyncIO within just a few Lambdas - heck even one Lambda could certainly handle hundreds, if not thousands of ASYNC jobs at once, just as a cheap machine, as you pointed.
They treated Lambda functions as threads or async jobs, while they should be looked at as core processors.
I'm not sure I understand the point about asyncIO: yeah, the Lambda CPU isn't spinning, but don't they count actual runtime? So even if things are async, the Lambda still needs to be up to respond to the event when the async IO is finished.
Or do you mean handling things in an async way and Lambda passing the info on to another computing environment?
I think what they're talking about is using one Lambda to persist, handling many crawl requests. That doesn't make a ton of sense, as each Lambda has to make sure to push all its results and shut down before the max timeout is up, but it would technically work and be more efficient than what I described. Much better, though, just to use long-running spot instances, though, or just a regular instance running something like Scrapy.
Exactly. These people thought they were using the tech well. They weren't. I like stories of tech gone wrong as examples to learn from. I thought others might as well.
Same result would be achieved by paying tons of $$ for a high-end machine with dozens of cores, just to scrape one URL per core.
They could have implemented asyncIO within just a few Lambdas - heck even one Lambda could certainly handle hundreds, if not thousands of ASYNC jobs at once, just as a cheap machine, as you pointed.
They treated Lambda functions as threads or async jobs, while they should be looked at as core processors.