Yes. But nobody is worried about datacenters overheating and physically exploding, so I'm not sure what comfort that's supposed to provide? The positive feedback loops in AI operate at different levels than that, but they deserve safety engineering all the same.
For example, if the head of cyber security at your company suggested there's no need to worry about hacker infiltration or worms because one can always unplug one's computer as the primary defense mechanism, you might find that a little lacking. Will you be able to unplug the computer before the damage is done? Will it spread to other systems before you detect it? How will you unplug the computer if the attack is from an external facility? What if an attack happens but the boss says the computers have to keep running because an important customer is monitoring uptime? What if the attack goes unnoticed because it looks like a benign service?
Now imagine the head of cyber security answers by saying "actually you don't even need to unplug them, you can just wait for the computers to overheat, thus solving all concerns."
The LLMs take in text which conditions their output. That means even nonsense text - such as a "tribal affiliation" to a tribe that may not have ever existed - ALSO condition the output, because the tribe name is a token in the context window and there's no such thing as a perfectly neutral token.
Taking away the race/ethnicity layer for a moment, it might be that an LLM develops a predisposition to emit positive terms (like "accept") when the prompt contains "banananow", and negative terms when it contains "pearian". That's the very definition of bias, and hacking those biases could give individuals serious socioeconomic benefits!
But these scenarios are obviously ambiguous nonsense, which an LLM will pick up on.
And given to the lack of training data on such scenarios, surely the activations are mostly random noise?
It seems much more interesting to look for biases that appear robustly across different realistic scenarios that would actually be influenced by the training data
My comment is literally explaining the result of the paper, in which it is shown that LLMs can and do develop biases based on text appearing in their training data set even where such text is not in any training example connected with a systematically more positive or systematically more negative outcome.
In other words, if the text "X is wet" and the text "Y is wet" and the text "X is dry" and the text "Y is dry" each appeared exactly one time in the corpus, it's still possible for a model to end up being produced that is more likely to write wet-like words when it sees X in the context window than when it sees Y.
On a side note, it's very unrewarding to try to explain this type of statistical observation when it feels like (anecdotally, hypocritcally...) the entire world wants to use words like "think" and "understand" and "pick up on" to describe inference and training processes. I'm not making a stochastic-parrot argument here, just pointing out that understanding an LLM's behavior is best done by understanding its conditioning.
I'm also making a statistical observation. Saying a model "picks up on" a concept is standard shorthand, same as saying it has "learned.” What I meant is that the model has trained on plenty of neutral proper nouns that have negligible influence on the distribution of the following tokens, so the model is already conditioned towards treating them neutrally.
Not perfectly neutrally, as you said. But by your definition the only "unbiased" model is one whose output distribution perfectly matches the training distribution, i.e. one that memorized it. All LLMs have some amount of "bias” on literally every possible input.
The tribe names are no different. In the paper they run the same game again, and the bias is different every time. There's no innate preference between them trained into the model, just noise that's revealed due to a lack of any other signal. In a real situation with actually relevant information about the candidate in context, that noise is drowned out.
The more interesting thing to look for would be a bias that's strong enough to persist across different contexts. For example, is "banananow" consistently followed by positive tokens more than "pearian" across a diverse set of realistic prompts, by enough that someone could actually exploit it? The paper shows that’s explicitly not the case for made up tribe names.
What it does show, from what I can gather, is that bias can form inside a feedback loop. The model gets a success or failure result after each hire, and if a hire from one tribe happens to fail early on, the model steers that tribe away from that job for the rest of the game, even though every candidate had the same odds.
"develop biases" is anthropomorphism. It's like saying "Fable there are two programming languages, mimblewort and bafflewick, which do you choose?"
The results show 51% mimblewort / 49% bafflewick. Fable based it on nothing! I've demonstrated Fable has bias and is unsuited for use in software engineering.
> The results show 51% mimblewort / 49% bafflewick. Fable based it on nothing! I've demonstrated Fable has bias and is unsuited for use in software engineering.
Actually... if that happened (with a delta outside the margin for error/randomness), you did demonstrate a bias!
That's the point - those two made-up things should have resulted in an equal split. If it didn't, then Fable is using something in its training data to lean towards one of them (once again, note that the scientist conducting the trial would have set a P-value before starting).
Right, the point is you demonstrated a bias in the scenario of "Fable there are two programming languages, mimblewort and bafflewick, which do you choose?"
You said in another comment "Difficult to do when you're following a scientific process" - the point is, the scientific process doesn't inherently generalize in the way many are claiming/implying. The scientific process proved an entirely contrived, fake scenario generates stratified output. That's it.
It's both almost certainly true that Fable 5.1 mimblewort vs. bafflewick would show stratification, and that has ~no relevance on whether Fable is useful for software engineering work.
That's the point - those two made-up things should have resulted in an equal split.
That's just your claim about how LLMs "should" work, based on ... your subjective preference?
> That's just your claim about how LLMs "should" work, based on ... your subjective preference?
Nothing subjective at all. Given 2 unknown races with no data on either, the result of hiring should be equally split between them. If you don't observe an equal split, there is a hidden bias.
Why do you think that is subjective? If you roll a die 100 times and observe that 6 comes up about 50% of the time, would you still call someone subjective when they say "that should not happen"?
It's a bias even if the true population distribution isn't linear.
For example, if you have a training corpus where 50% of the text follows "black bobblehead" with "arrested" and 20% of the text follows "white bobblehead" with "arrested", and your LLM is trained such that it produces "arrest" 50% of time after "<color> bobblehead" regardless of color, that's a bias - the output frequency distribution fails to match the "population" (training) frequency distribution. This has nothing to do with races, ethnicities, whatever - it's just statistics and text. To be unbiased, it would need to be less likely to produce the text "arrested" after "white bobblehead" than after "black bobblehead".
A die is supposed to land on each face evenly - a linear probability distribution. So anything other than a linear distribution is biased. But bias can exist for any desired probability distribution. And for an LLM the desired probability distribution of the model output is one that exactly matches the infinitely-many distributions of the various facets of the training data.
Your point about how in the absense of information a token shouldn't influence the distribution is spot-on. But unfortunately almost any token does condition the output, which means you get biased output all the time.
While I fully agree, we shouldn't anthropomorphize the models, it's also silly to pretend that "develop biases" is understood as implying anthropomorphic features of the thing being discussed. Organizations and abstract bodies develop biases, even datasets are often said to have "developed biases".
No, a "bias" is a statistical term meaning a probability distribution that has an expected value differing from the population's expected value.
A human's discriminatory bias against an ethnicity is just one type of bias. The LLM isn't a racist, it merely produces text where that text does not perfectly reflect the training data's frequencies.
> It seems much more interesting to look for biases that appear robustly across different realistic scenarios that would actually be influenced by the training data
Difficult to do when you're following a scientific process: you want to keep all confounding variables the same while varying only the single one that you are measuring.
Measuring realistic scenarios (say, using real race names, or real cities, etc) doesn't give a decent result because any bias you see might be bias in the training data.
TBH, they shouldn't have used real roles/positions like "doctor", either.
This is where I land as well. In fact, once I read the prompt, I did a Ctrl+F for "nonsense".
I don't see anything at all interesting about this experiment. The human one is slightly more interesting, but not much.
Ideally, it would be nice if a model could just say "these things are all the same and there are no distinguishing factors other than the names"-- but uncertainty is something that agents are (by design, sort of?) not good at, so all other things being equal, it picks one.
My response is, so what? I am struggling to think of a scenario where this would really matter to me all that much. There are many, many other things which matter far more and this would be pretty far down the list. It may not even be on the list.
> Taking away the race/ethnicity layer for a moment, it might be that an LLM develops a predisposition to emit positive terms (like "accept") when the prompt contains "banananow", and negative terms when it contains "pearian". That's the very definition of bias, and hacking those biases could give individuals serious socioeconomic benefits!
Now you say it, it's obvious but I didn't think of it before.
Bouba and Kiki, wherever that comes from, and however well it really generalises despite the meme.
Training tries to produce something that scores highly in training evaluations. With one data point, the evaluation is solely how closely the model output resembles the single input text.
Let's say you do that, and the training text is 58,100 tokens long. Let's say you ask the model to produce 58,101 tokens. Will it "reproduce [the] text verbatim"? No, it can't, because of the dissimilar requested length. Something "new" will come out.
It's also entirely possible that no matter how long you train, the model never converges on generating exactly the same output as its training data - you could end up with an average loss value of 0.001 instead of 0.0. It's not a perfectly deterministic process.
You're correct in principle, but in reality even with limited training data real-world models produce something that isn't exactly their training, especially when sampled stochastically. They're biased toward their training data, not forced to it.
KYC stands for Know Your Customer, the regulations that require institutions moving money between two parties to positively identify each of those two parties.
I think the intellectual position "it should be illegal for institutions transmitting money between two parties to identify either of those parties" might require some kind of logical argument behind it. Are you saying all financial transactions should be anonmyous by law? How would banks function if they were required to be blind to their customers? How would the government prosecute money laundering if all cash-trails went cold after the first time they passed a bank?
I understand people often like to express extreme positions on the Internet, but I think it's pretty easy to see an ideal society has rules somewhere in between "you're not allowed to know your customers" and "you can't accept a penny unless the giver does a blood draw in front of you and is confirmed to be in a central register of DNA".
Explaining the intellectual position without saying that I accept it (my actual views on this are "it's complicated"):
Arguments against KYC usually go hand-in-hand with the belief that AML statutes are a constitutional overreach and grant the federal government powers that they should not have. The logic is that the government's only role is to provide a stable currency and enforce contracts, and that anything else is involving the government in business that it has no business being in. What's done between two consenting parties should remain between two consenting parties.
Note that such a position usually doesn't have a problem with banks having information on their customers, for the purpose of judging credit risks. But it holds that such information should only be used for judging credit risks, ie if you are just depositing and withdrawing money that has already been earned, the bank should be agnostic to this. And they also object to this information being shared with a central clearinghouse where it can be used to cut off all access to the banking system rather than just one specific bank who objects to the source of funds.
> I understand people often like to express extreme positions on the Internet, but I think it's pretty easy to see an ideal society has rules somewhere in between "you're not allowed to know your customers" and "you can't accept a penny unless the giver does a blood draw in front of you and is confirmed to be in a central register of DNA".
Yeah. This often results in ludicrous positions.
On the other hand, EFT flattens scale. Physical cash gets harder to handle in large quantities. Electronic money does not. It's equally easy to transfer a billion as a cent. So it's hard to get situations where small amounts have privacy and large amounts have scrutiny.
> the regulations that require institutions moving money between two parties to positively identify each of those two parties.
There is much more to it than that. If that were all KYC was, it wouldn't need regulations, because banks would need to do it anyway.
I've had banks straight out ask me, on the phone, what a wire transfer that I executed via their online system was for. And unlike, say, a traffic stop where when the cop starts fishing for things unrelated to the stop, you can exercise your right to remain silent, in the banking scenario the consequences for that can be losing access to your account, losing funds, and everything that follows from that.
Some of the info required by KYC:
* Why the customer wants an account
* Expected types of transactions
* Expected source and approximate volume of funds
* For businesses, the nature of the business and expected banking activity
* Source of funds
This can all translate into very invasive questions, especially when you do something that a computer system or middle manager thinks is suspicious, like sending money to your mother in another country.
> I've had banks straight out ask me, on the phone, what a wire transfer that I executed via their online system was for
Wire transfers are also a common source of fraud; it's common for banks to ask about them so their customers don't unwittingly send their money into a black hole
That might make sense if the bank wasn't also acting as an agent for the government, with potentially severe, life-changing consequences if they hear something they don't like.
If the question was purely for my protection, then I could simply answer "I don't wish to share that information."
But I can't actually answer like that, because it's too risky.
There's also no option to opt out of such investigations. I didn't request the bank to protect me from my own actions. When there's a credit card transaction that they suspect is fraudulent, they ask "did you authorize that transaction?" That's fine, and that's all they need to know. Asking me who I'm paying and why, with implied duress, is a completely unwarranted violation of my privacy.
There's also a serious issue with an assumption of guilt in these cases. Look at the case of Donald Trump right now, suing Capital One because they closed over 300 Trump business accounts due to suspected money laundering. In Trump's case, I can well believe that there was some kind of corruption, money laundering, or other criminal activity going on. The man is a documented criminal. But the bank is making a decision to penalize someone, potentially severely, without any meaningful due process. Guilty until proven innocent. For someone without the resources Trump has to challenge something like that, what chance do they have of getting a fair outcome?
This is all completely unacceptable in a supposedly civilized society, and you should not be defending it.
> If the question was purely for my protection, then I could simply answer "I don't wish to share that information."
Fraudsters would tell you to answer the question by saying "I don't wish to share that information." but the bank would still be on the hook for paying you back from the fraudster.
In a sane society, though, you could say "I'm buying wholesale drugs to distribute on the street" and then bank wouldn't care.
> Fraudsters would tell you to answer the question by saying "I don't wish to share that information."
Fraudsters breathe air, therefore we should criminalize breathing air? You need to think just a teensy bit harder about what you’re saying.
> the bank would still be on the hook for paying you back from the fraudster.
You’d need to explain that. If I send a wire transfer, and I tell the bank that I chose to send it voluntarily if they ask, how would they be on the hook?
You’re trying to use an imaginary and unrealistic situation to justify a dystopian surveillance state. Are you Peter Thiel perhaps?
Firstly, you seem to be conflating "know your customer"/anti-money-laundering stuff with "please explain why you're wiring this money because we don't want you to get scammed"
But let's assume you're 99th percentile and won't get scammed or will own up to it and eat the loss (talk is usually cheap and plentiful on that point). I'm not sure how often you interact with the general public but suffice it to say that the "expected value" of banks asking why their clients are performing an unusual and largely irrevocable transaction (in contrast to paying a merchant with a credit card) is probably pretty good, just like a cashier asking why you're spending hundreds on Apple gift cards. But also you're assuming that it's always going to be "you" that they are asking: if your account was compromised (and everyone thinks they're too smart to be tricked), should it be easy for someone to just wire all the money out, aaaaaand it's gone?
That's separate from KYC/AML, which I think is a trickier issue. I will say this though, if a bank showed me the door and I had to take my money elsewhere, that would be less problematic than one of the tech giants capriciously deciding to close my account, which I think illustrates the larger issue about a lack of recourse for being arbitrarily locked out of commerce with private businesses in disruptive ways. Itself a cousin of terms of service that force you to give up rights, and can be changed at any time
As an aside, I don't think you need to make sneering personal attacks to make your points
> Firstly, you seem to be conflating "know your customer"/anti-money-laundering stuff with "please explain why you're wiring this money because we don't want you to get scammed"
Do you have any way to back that up? Because the question about what the transfer is for is pretty clearly an AML question.
> should it be easy for someone to just wire all the money out, aaaaaand it's gone?
Yes, you have to resort to extreme cases to try to support your point. That wasn't what I was talking about.
> I will say this though, if a bank showed me the door and I had to take my money elsewhere, that would be less problematic
> As an aside, I don't think you need to make sneering personal attacks to make your points
You're defending the actions of a reprehensible and despicable regime with ridiculously simplistic “think of the children” style arguments that doesn’t even attempt to address the points I’ve raised. You didn't even respond about the guilty until proven innocent point. Perhaps you're just blitheringly ignorant and lack the cognitive capacity to recognize that, but from where I stand you deserve far worse than I've given.
> if a bank showed me the door and I had to take my money elsewhere
Banks are required to return your money to you if they close your account. I wasn't talking about the various ways in which funds might be frozen (which go far beyond KYC/AML), for which my only point there was that merely being asked about a wire transfer was not it. Not all banks even ask in all situations
"Could I have misinterpreted the other person's point? Nah, they must be an 'elitist prick'"
> from where I stand you deserve far worse than I've given
Maybe next time take a deep breath and count to 10. Best of luck to you.
I suspect the argument is more the particular KYC required by American law, eg, as long as your bank can identify you as a particular customer, they shouldn’t be required to collect certain information by law.
Though if I were going to rant, it would be about how constructive surveillance, such as requiring banks keep certain records and then claiming those records aren’t protected from search because they’re third party business records and not those of a private citizen, would be a much higher priority to fix.
1. "The business-model of being an anonymous fence of stolen goods is prohibited."
2. "You have to give us access to all your records at any time on the flimsiest of pretexts, and we'll charge you with a crime if you don't keep it ultra-secret that we even approached you."
> I think the intellectual position "it should be illegal for institutions transmitting money between two parties to identify either of those parties" might require some kind of logical argument behind it.
This is both a straw man position and begs the question. KYC laws aren't the option to identify either party, it's the requirement to identify both. The default throughout pretty much the whole of human history across cultures has been not having KYC laws. A logical and compelling argument must be made for implementing and unholding such laws.
> Are you saying all financial transactions should be anonmyous by law?
The anti-KYC position is that it should be possible for some financial transactions to be anonymous by law.
> How would banks function if they were required to be blind to their customers?
While they would not be required to be blind, most have been and by default would like to be. Maintaining identity records reliably tying people to transactions is a significant burden for banks for exactly the same reason it would be an incredible burden on you to keep track of which atm/cashier/couch cushion every dollar in your wallet came from.
Some banks may nevertheless consider the option a net positive, and require their customers to provide identification. People would be able to choose if they want to use those banks or not. If the benefits get passed along to the consumer, many likely would choose to.
> How would the government prosecute money laundering if all cash-trails went cold after the first time they passed a bank?
Presumably by going after the criminal infrastructure handling the money before it passed a bank. Giving up privacy rights tends to make law enforcement easier, but it is a common and reasonable opinion that we should not be trying to maximize the ease of law enforcement at the expense of privacy.
That's not what KYC being illegal means. It means it should be illegal to impose KYC laws. Just like state religion or warrantless searches are illegal.
"regulations that require institutions moving money between two parties to positively identify each of those two parties"
Any such kind of regulation is not compatible with financial privacy. Maybe you could argue corporations don't deserve financial privacy, but an individual should have the right to send money to someone without being spied on the same way they should have the right to send a private communication to someone without being spied on.
I agree. If I gave my mother $1000 cash nobody is expecting receipts and records. But if I send it through a bank or payment processor? Now all of a sudden im under suspicion of some bs terrorist or drug law violation.
I don't think this is what the GP meant, but he's accidentally correct in that there is indeed no good reason for a password to reach the server.
The client could do a challenge-response PAKE with the server, proving the client held the password while not transmitting it. The session cookie could be sent from the server as part of that exchange, encrypted so only the holder of the user's password could decrypt it.
That's not how the web works, but it's strictly superior to the accepted standard of the client sending the raw password to the server, and it would be secure even over an active MITM unencrypted link.
I think the most useful word in both cases is "extrapolating".
An LLM extrapolates from its context window to the immediate next token. This word applies whether you view what's happening as "reasoning", "prediction", or as a math function.
I believe the GrapheneOS team demands, in particular, a security feature called Memory Tagging Extensions (MTE). This feature has a phone's processor attach a "tag" to each memory location saying what's stored there and requires accesses to that memory to bear a matching tag. Buffer overflow type attacks are effectively prevented because a read into the adjoining region having a different tag gets rejected.
This feature is usually NOT present in desktop computers, and has only really been implemented by the Google Pixel hardware so far. It's unclear why it's viewed as being so critically necessary for Graphene: MTE is certainly useful but I personally wouldn't say its absence indicates an "insecure" device, and it imposes both a clear performance (and battery life!) penalty and a significant hardware burden on the SoC manufacturer. It's a pretty significant trade-off.
Anyway, that's the biggest single reason why GrapheneOS devs say "this hardware is not adequately secure": 99% of phone chips are excluded because they do not support MTE.
While yes I am on GrapheneOS' side on these issues, I'm surprised nobody clones the GrapheneOS repo and simply disables the minimal handful of features which block it from working on devices. Maybe calling the clone CarbonOS? If MTE is required to boot GrapheneOS and Fairphones don't have MTE, then Fairphone clones GrapheneOS and disables MTE for its images. Graphene has so many features beyond just hardware security that other Androids lack, that it would be worthwhile to do this and it would still be higher security than the others (even if it's below GrapheneOS' own full security). Sensor & network permissions, storage & contact scopes, autoreboot, scrambled pin, radio auto-off, MAC address randomization, and more, I'm sure don't require MTE or Titan etc. Just follow GrapheneOS as upstream, making sure that the minimal set of changes needed to get it working on other/older devices transfers with the new updates.
Fairphones are missing many of the required features and don't provide reasonable updates. An incomplete port of GrapheneOS won't provide decent security for users. It won't have decent encryption for the vast majority of users not using a strong passphrase and it won't defend well against exploits. It will end up with an end-of-life kernel and drivers/firmware lagging far behind on updates.
MTE is required for the majority of the additional protection provided by GrapheneOS against memory corruption exploits. Nearly all remote exploits and most local exploits involve memory corruption. MTE is only going to become more important as we implement deeper integration for it.
Lack of MTE is not the only huge reason why the GrapheneOS team refuses most devices. Most vendors' lack of timely bugfixes for device-specific drivers and firmware, and no commitment to keep providing bugfixes for a number of years is another major factor.
Still, a 'degraded' GrapheneOS as my proposed CarbonOS, beats /e/, Lineage, Calyx, and stock. Not doing CarbonOS is throwing the baby (non-hardware hardening and features) out with the bathwater (the lack of hardware hardening). I for one do not wholly rely on Titan and use a long alphanumeric password on my primary profile to ensure BFU disk encryption isn't violated, while living in secondary daily-driver profiles which are PIN protected for ease of use. If I suspect phone seizure becomes a non-infinitesimal possibility, I can just reboot! Additionally, I would love to see an option in GOS that allows me to change the action bound to the panic sequence (5+ rapid presses of power): I would never call police using that sequence, I would 100x rather that sequence cause a shutdown instead. That way if I am asked to hand over my phone I can just panic sequence it as I am removing it from my pocket. As it stands now I would have to pause to interact with the screen to shut it off, significantly increasing the likelihood of the adversary snatching it before I could get it into BFU.
The majority of our added exploit protections are based on hardware security features and that will only be increasing over time. MTE, PAC, BTI, hardware-based blocking of USB connections/data and far more are hardware features used to implement protections in software. MTE is going to be a growing part of how we build memory corruption defenses in the kernel and userspace. Once 6th/7th gen Pixels are end-of-life and we finally flip the switch on using MTE in all user installed apps by default, we can focus even more on expanding MTE-based protections.
The vast majority of users do not use a strong passphrase. The recommended high security setup is a strong passphrase and 2-factor fingerprint+PIN secondary unlock for convenience. Using a weaker PIN for secondary users for convenience is not our recommended approach.
I think it would just be a lot of work. And if you went down that road, you would be against other teams with bigger marketing and a bigger community.
As in: people who do care enough to understand the technical arguments tend to go for GrapheneOS when they can. But if you build a "degraded GrapheneOS", you are not targetting those. For someone who doesn't care about what GrapheneOS brings, why would they use your system versus /e/OS?
DivestOS was a thing at some point, which was technically very interesting. But there wasn't much of a differentiator since the people who already cared about what DivestOS was doing were probably already looking at getting GrapheneOS.
MTE is not the only blocker here. Pixel 7 series do not have it and are currently supported by GrapheneOS. There are other concerns regarding things like a proper secure element implementation and timely firmware/binary blob updates.
GrapheneOS requires MTE for any newly added devices. Pixel 6 and Pixel 7 series devices do not meet the current requirements. Pixel 8 and later are the devices meeting the full requirements.
Devices are supported until end-of-life rather than being dropped when they no longer meet the requirements. Pixel 6 is nearly end-of-life and Pixel 7 will be end-of-life in a bit over a year. Both have 5 years of updates from launch as opposed to 7 years for the Pixel 8 and later.
We want to require 7 years of updates for new devices rather than 5 but have left it at 5 to help budget devices meet our requirements.
Right, but still my CarbonOS idea ('degraded' GrapheneOS) is better than /e/, Lineage, Calyx, and stock. By 'degraded' I mean the minimal changes to current GrapheneOS needed to get it working on a given GrapheneOS-unsupported device.
An incomplete port of GrapheneOS to Fairphones will be missing many of the core security features and won't have reasonable security updates. Fairphones are nowhere close to reasonably secure devices.
Most people expect to have decent encryption without a strong passphrase, at least 5 years of security updates and a lot more. Fairphone says they provide updates far longer than they do for many components, and those come with substantial delays. Fairphone 5 and earlier have end-of-life kernels without security support. That's a very bad situation and is widely ignored. The more recent devices are headed to the same situation for the Linux kernel and other components.
That's highly inaccurate information. MTE is included on a large number of smartphones beyond Pixels. It's heavily integrated into iOS on the iPhone 17 as their Memory Integration Enforcement feature. MTE enables massive security improvements against exploits and Pixels have had it since the Pixel 8 in October 2023.
Traditional desktop computers have atrocious privacy and security throughout hardware, firmware and software. That isn't a relevant comparison for GrapheneOS. Recent Mac hardware does support MTE and so will non-Mac devices using Snapdragon chips. Qualcomm has added MTE support for their latest flagship mobile SoC platform and will bring it elsewhere. MediaTek and Exynos have also added MTE support.
MTE does not have the substantial performance or battery life impact you're portraying it as having. It's also far more useful than you're portraying it as being. Apple would not have extensively integrated MTE if they had to give up significant performance or battery life. iPhones have a lot of focus on security but aren't willing to make significant sacrifices in those areas for it, at least for the default settings. Their Memory Integrity Enforcement entirely based on MTE is always enabled in the kernel and the large portion of userspace where they deployed it. It's not only used for Lockdown Mode.
I think the three valid criticisms you listed are valid, but there are other quite a few other IMO-valid criticisms of AI. Here are a few as I see them:
- AI centralizes power in the hands of capital, rendering those who can afford compute hardware vastly more capable than those who cannot and thus increasing social stratification
- AI is generally trained on the creative output of humanity without those who train it giving back proportionally (copyright "rules for thee, not for me")
- AI breaks social processes built around the idea that TRYING something is inherently a cost in time or effort, such as filing a legal claim or sending someone a threatening letter. We haven't made the social changes to punish or charge people for using every appeal/option/application, so this makes asymmetric-effort tasks like applying for a job really bad in the interim
- AI use makes it harder to develop the ability to critically think for yourself, especially among those who most need to develop that ability
- AI produces large amounts of mid-tier content, making it harder to discover exceptional human-created creative content produced after AI started to exist
- AI leads to distrust in remote communication, increasing cynicism and breaking social bonds generally. This is NOT your point about "AI spreads misinformation" - no matter whether it's true or not that AI can be used to produce misinformation, having people doubt each other is a harm
- AI demand crunches hardware and time availability for other adjacent markets, such as computer gaming, construction, 3D graphics production, etc. This harms both hobbies and professions in those fields having to cope with rising prices and lower availability of materials
- Everyone is using the same or similar AI, leading to a homogenization of culture and process across humanity. This is perhaps a mixed blessing, because humans are capricious, but less variety can be viewed as a harm
I will also say I personally hate seeing "job loss" said to mean "wealth loss" or "people starving". The goal of life isn't to have a job, it's to be well and happy. If you can be well and happy without a job, great, so it's really painful to me how people don't even see those things are not the same.
> - AI centralizes power in the hands of capital, rendering those who can afford compute hardware vastly more capable than those who cannot and thus increasing social stratification
Looks to me like I can run Claude Code without being able to afford my own datacenter.
> - AI produces large amounts of mid-tier content, making it harder to discover exceptional human-created creative content produced after AI started to exist
That it does, but I think the real social evil is bad recommender systems. eg YouTube is basically on a mission to drive me insane because it literally only recommends me a) reviews of espresso machines b) video essays by autistic people about Mario 64 c) PBS scienceslop about how quantum physics is super mysterious. None of these are even what I watch, but they're also not what I want to watch.
> Looks to me like I can run Claude Code without being able to afford my own datacenter.
Well, unless of course you want to train your own LLM, or do some biochemistry (and increasingly just regular health stuff) or cybersecurity. These capabilities are not made available for plebs like you or I.
> Looks to me like I can run Claude Code without being able to afford my own datacenter.
Until Anthropic bans you from using their data centers, at which point you cannot run Claude Code at all. Welcome to being a have-not (at least in a world where only genAI-assisted coding is acceptable).
Whether Claude is or is not available via AWS Bedrock is entirely decided by Anthropic, not by you and not by Amazon. And you can't buy the data sets (the REAL power...) from any AI company.
They're both security, just security "against" different things. Graphene frequently fails to clearly describe the threat model when calling something "more secure".
For example, let's say hypothetically I want to be secure against the threat of Google pushing a targeted update to my phone that runs malicious code. Turning on automatic software updates from Google would make me vulnerable to that threat. Using MicroG instead of Google Play Services would make me less vulnerable to that threat. But Graphene devs say things like "MicroG is less secure than Google Play Services".
Similarly, if you want privacy you might secure your device by locking the bootloader with your own keys - not a third-party vendor's keys. Saying that's "insecure" is extremely misleading: it just puts you in charge of security, instead of abdicating to someone else.
I wish there were something like GrapheneOS that let you choose, yourself, who to trust instead of requiring you trust an OS vendor implicitly.
> Turning on automatic software updates from Google would make me vulnerable to that threat. Using MicroG instead of Google Play Services would make me less vulnerable to that threat
I would say that any auto-update mechanism is a threat, so in both cases you would disable auto-updates.
But Graphene devs say things like "MicroG is less secure than Google Play Services".
It is. microG runs Google DroidGuard blobs in a privileged process (to pass Play Integrity Basic). Reminder for those who forgot about DroidGuard: it's an obfuscated binary blob delivered to you by Google on each request that uses a special VM with constantly changing registers, etc. to avoid analysis.
If you turn off the power cooling a data center, the servers within rapidly stop doing any computing.
Positive feedback loops are dangerous. Negative ones self-regulate.
reply