While I was asleep, apparently the site was hacked. Luckily, (big) part of the lemmy.world team is in US, and some early birds in EU also helped mitigate this.
As I am told, this was the issue:
- There is an vulnerability which was exploited
- Several people had their JWT cookies leaked, including at least one admin
- Attackers started changing site settings and posting fake announcements etc
Our mitigations:
- We removed the vulnerability
- Deleted all comments and private messages that contained the exploit
- Rotated JWT secret which invalidated all existing cookies
The vulnerability will be fixed by the Lemmy devs.
Details of the vulnerability are here
Many thanks for all that helped, and sorry for any inconvenience caused!
Update While we believe the admins accounts were what they were after, it could be that other users accounts were compromised. Your cookie could have been ‘stolen’ and the hacker could have had access to your account, creating posts and comments under your name, and accessing/changing your settings (which shows your e-mail).
For this, you would have had to be using lemmy.world at that time, and load a page that had the vulnerability in it.
There’s a huge difference between alienating people in an effort to make the website profitable, and outright paying hackers to go after Lemmy, which would be a criminal action. Let’s not get all big brain on here and pretend Lemmy.world is a threat to Reddit and is pulling away tons of users, to the point that Spez needs it out of commission. I didn’t jump here from Reddit to promote conspiracies
Very impressed by how quickly action has been taken by this and other instances to patch the issue.
Very, seems like great work.
uh, why did you have negative one dislike?
Negative one upvotes would mean that enough people disliked me/another poster to bring my upvote total to zero. (Upvotes and likes are effectively the same thing, it’s just a naming convention). Reddit totals them up and seemingly Lemmy does as well.
huh that’s weird (yes I meant negative one downvote), I already know that the total can be either positive or negative, but shouldn’t the upvote number and downvote number be either positive or zero? (for now I’ll just accept it as a lemmy bug/ inconsistencies between instances)
Nope, just like Reddit it’s a value that ranges between negatives and positives. If I get two thousand upvotes, positive 2k. If I get two thousand downvotes, negative 1999 (because iirc you start with one by default).
Not exactly sure I understood what you meant by “either positive or zero”.
see your comment rn, it has 1 upvote (from yourself by default) and 0 dislike (so it’s not shown)
but in the screenshot I sent above you got 287 upvote and minus -1 downvote (making your total 288) which is mathematically correct but seems like an unintended behavior
for example this comment of mine normally have 9 upvote and 2 downvote (which is shown as a positive integer 2, not negative), making my total upvote 7
Just occurred to me that the app I use also shows separate counters. I fooled myself into thinking it was a single counter.
That’s interesting. Remember it’s a very new platform, minor bugs aren’t out of the ordinary.
Hijacking the top comment to say I had problems with logging in to Lemmy.world today and liftoff was failing in odd ways.
I had to go into my web browser and clear my site cookies for lemmy.world to let me log in there.
In liftoff I had to go into the app settings in android to clear the cache and then remove and re-add my account for it to be able to log me in. (Press and hold on the account to remove it)
In liftoff I had to go into the app settings in android to clear the cache and then remove and re-add my account for it to be able to log me in. (Press and hold on the account to remove it)
Good PSA. It took me a bit to figure it out, the app doesn’t make this obvious.
I’m on iOS with the Memmy app. It’s a work in progress that’s officially unfinished so I’m not surprised but it has also been a bit buggy. Doesn’t seem that I can log out without deleting and reinstalling the app so hopefully this doesn’t happen too often XD
Go into account settings, clear your password, re-enter your password, save, go to feed and pull to refresh. That’s what worked for me.
So I was actually just struggling with that myself, also in the Memmy app in case that isn’t clear
What I did was add my account (again)
There was no warning or anything, and it populated the list with two of me.
At that point, a “delete account” option appeared under both of them. So I guess in normal circumstances, it wants you to keep one account around at all times?
I deleted one of them, and the app basically reinitialized. Both were gone and it showed me the welcome screen.
I logged back in, and now everything is back to normal
Finally I found good instructions, was about to delete and reinstall until I followed this!
Interesting. Definitely could be made clearer, I’ll make a post on the GitHub later about some of my suggestions.
Whoops, glitched double response.
Ah interesting. I’ve had multiple accounts from the start so it was much easier for me. Just removed my main account and added it back.
I did this, but I just didn’t delete either accounts and it worked fine. Idk if it’s detrimental to have two of the same but it worked for me.
I just did edit account and then saved, it seemed to trick it into logging in again (secrets on my instance were also reset).
Interesting. Definitely could be made clearer, I’ll make a post on the GitHub later about some of my suggestions.
For Memmy, I went to the accounts page in the settings. Click d on my lemmy.world account then to the page where you can change the password then navigated away. That reactivated the account. Maybe we should add a ticket on Memmy’s GitHub about reactivating cookies when there’s an issue. Or at least place à poput to double check credentials or something.
I found I didn’t actually have to log out, just go into account settings and reconfirm everything without changing it
No you can. You just remove the account from the accounts list. It’s labeled “delete this account” which is scary but it just removes it from Memmy. You can add it right back and that logs you back in. Not a great experience.
I sure hope this doesn’t happen a lot. This kind of barrier hurts site growth. I’ve managed a lot of large sites and seen a lot of bugs and when everyone gets logged out there is a measurable impact, and some folks never return. Just look at all the comments here saying “thank I didn’t know to do that.” For every one of those there are 100 people going “huh… Lemmy is down… oh well… on to something else…”
I was I able to upvote anything or subscribe. Seems like uninstalling and reinstalling fixed my issue
thanks for posting this, I wouldn’t have figured that out lol
Oh, I was wondering why it was showing me as logged in but wouldn’t let me upvote due to not being logged in. Your liftoff psa just cleared that right up for me, thanks!
Thank you so much. I was having the same issue with the website and PWA. Clearing site settings worked perfectly.
How have I never thought of comment hijacking?!
The XSS fix was part of UI version 0.18.2-rc1. Lemmy.world is running 0.18.1.
Are we sure it’s fixed?
the details of the vulnerability are already known now anyway since there’s a fix that was proposed on the Lemmy GitHub so I don’t think it will hurt others to talk about it
Could you please link the issue? Thanks!
https://github.com/LemmyNet/lemmy-ui/pull/1897/files found it myself
Oofof… That’s not suppose how we announce vulnerabilities…
yup that’s the one
what I find weird is that the “fix” still focuses only on the front-end, the issue is still that unescaped HTML is being stored in the database and still trusting the front-end is nuts
I mean, I’m pretty sure that for an XSS attack that’s fine. The entire problem is that somebody posts e.g. a comment that contains code that is automatically run in users’ browsers. If you make the front end just not execute that code then it’s fine. Who cares what’s stored in the back end?
I mean, it would still be better to have multiple fail-safes, and they probably should still sanitize text entering the database.
But this is sufficient for a quick fix.
Let me introduce you to my friend, Little Bobby Tables… :)
ALWAYS SANITISE!
For sure it is sufficient for a quick fix. But a Lemmy post can be posted not only on Lemmy but on other front ends (like kbin, mastodon, and many others) and they can suffer from a similar attack due to the backend storing and forwarding the bad content. So, it should not be stored as it is in the backend
I think people are forgetting that it’s somewhat obvious the hackers or whomever, I don’t really care honestly are Lemmy users considering they did this at night and got into the site so quickly to begin with, they’d have to have been familiar with it to get into it as fast as they did.
If anything everything should be fixed.
I think it makes sense to escape as close as possible to the context where the data will be used, see https://benhoyt.com/writings/dont-sanitize-do-escape/
I think the main developers are aware of either of them but I’m not sure, haven’t seen anyone site admin wise talk about this mess.
This discussion on the original bug report does talk about the back-end needing a fix as well.
The devs are aware
May I ask was is the JWS coockinand if it is automatically changed or if we have to change it in a way?
If I’m not from lemmy.world and visited a lemmy.world community via my home instance, does the hacker gain access to my account?
If I, while logged in to my home instance, accessed lemmy.world in another tab, does the hacker gain access to my account?
Does this hack infect devices used to access lemmy.world?
Sorry for noob questions, I’m just worried.
I am still not sure about it, but if a compromised comment reached your instance (through federation) and users in your instances viewed that comment, they have been hacked too.
MAYBE you are safe If your instance has no custom emojis enabled.
afaik, exploit does not pass through federation. but you should change your password just in case.
it doesn’t and probably cannot infect your device
👍
deleted by creator
Thanks.
Thanks for the quick reaction and TRANSPARENCY!!
I heard that some instances were defaced. Any examples of this? I wasn’t online this noon so I never got to see any action.
Damnit, spez.
Is there a rough time range when it happened? and any news about other big instances like lemmy.ml? Are those safe? Currently they are not on the same version as lemmy.world.
2:11 UTC is my first record of the event taking place, but keep in mind the attacker could have injected code long before without noticeable impacts. There’s no way to be completely certain they didn’t steal tokens and access accounts before they made themselves known.
Is a password change advised? How does the JWT cookie and exploit effect apps eg Jerboa?
You will have to login again for those apps. As far as we know, the exploit doesn’t allow someone to actually steal your password directly, just the session you were logged into.
However, it is my personal opinion that you should change your password anyway out of an abundance of caution.
Is that why Liftoff wasn’t loading?
Is this why I had to sign in and out of my account on liftoff?
I couldn’t comment untill I did that. There may be others!
I had a similar issue where my subscriptions were blank. A logout and re-login fixed it. Thanks.
