a random collection of whatever

an adult and I tag everything

avatar from kidd.waterfall.social


Posts tagged waterfall:

Stupid Bits From Waterfall's Code Over The Last Eighteen Months

thelldev -

I'm going to be honest, I know you're meant to use title case in titles but I don't actually know which words you're not meant to capitalise

I was dealing with a bug report in the Discord earlier and it hit me - all currently known bugs are artefacts of some code I wrote over a year ago and shoved in a file, never to be touched again.

Or, well, no, that's a little bit of an exaggeration. But not much - working on Waterfall has taught me a lot over the last year, so I thought, while I'm working on the API rewrite, it might be amusing to go over some of the more stupid bits of code I've written, the bad decisions, and just things that made me laugh.

All the dumb shit here is fixed in the API rewrite.

1. Have I reblogged this?

The bug report that spawned this post is this one, actually. Someone knows they've reblogged something, but the button never turns green. Annoying, since, other than XKit, we were the first Tumblelog site to implement that feature to my knowledge. Let's look at what could be the problem, shall we?

Notes are stored in their own table in the database, and it's currently over a million entries long. If something is meant to be displayed to a user, it comes from this table. In the case of detecting a reblog, it searches a combination - has [blog ID] reblogged [post source ID]? If yes, it'll be green.

The problem here arises from the simple fact that the notes table is not expected to be 100% consistent. In fact, it's the lowest priority part of the site - while obviously it's not desired, it's considered that if some notes don't get logged, that's fine. Avoid it if possible, but if it happens, no big deal. All the data to reconstruct the table is elsewhere anyway. We could nuked it and have every entry regenerated in less than an hour if we needed to.

And there's the problem. Searching the notes table is inefficient and stupid, especially since it's not consistent. What we SHOULD be doing is searching the Posts table, and asking whether that blog has reblogged a post with the same source post. About 5x faster, and significantly more accurate.


NSFW tags are mandatory. Always have been. Side ramble: I've started seeing a couple blogs putting "I define NSFW as..." in their blog descriptions. No, you're wrong, go and read the rules again.

Anyway, the code, right now, has hardcoded checks for specific tags. When we overhauled the tag system a few months ago, we pre-seeded two tags - "nsfw" and "NSFW". These are tag IDs 1 and 2 respectively (others were added to the database in the order they first appeared in posts - so the first actual tag on the site has an ID of 3). If you're logged out, a minor, or just have adult content turned off, the site runs a separate query to get posts for you that deliberately excludes posts tagged with that. Over time, the query has been amended to also include tag 938 in the query, because someone on mobile did "Nsfw" at some point.

You can see where this is going. Every tag variation, it needs to be added manually to the queries. The same is true of DNR and DNI tags - variations in casing keep appearing and have to be added.

This doesn't go into people who use the tag "nsfw gif" or some such, and not the plain nsfw tag - those are completely omitted from the queries and not caught. There's a really, REALLY obvious solution I missed here, and I'm upset it took me so long to realise it.

When the post is loaded, just. Have it check the first four letters of all the tags and see if they match "nsfw", and if it is, skip it and get another one if appropriate to do so. It's so simple. I'm so dumb. Writing this I realise I haven't gotten it set up in the API rewrite to use the same tactic for DNR and DNI yet. I'm an idiot.

3. Upload Limits

This was just dumb on my part. Most people think of megabytes as being 1000 kilobytes. In actuality, it's 1024 kilobytes. A gigabyte is 1024 megabytes, and a kilobyte is 1024 bytes, etc. When I added the upload limits, I completely forgot about this fact, as well as, apparently, the fact that I have a degree in computer science, and used 1000 as the figure.

This is fixed now, but there is still an issue relating to uploading stuff near the limit that's probably some weird base64 encoding issue. Oh yeah, speaking of.

4. Canvases

I hate Javascript. Seriously, it's terrible, I hate it. But I have to use it and not doing so is costing us users, which is why there's a big rewrite going on. The biggest JS component on the site right now that I wrote myself is the post editor for image/art posts.

This particular bug related to when you're editing a post rather than making a new one. If you uploaded a gif in the original and go to edit it, the GIF plays normally. But when you hit save, it turns into a static image at whatever frame it was on when you hit save.

When uploading a new file, it doesn't really need changing - you have the file, so it can just send the "blob" (that's a technical term, honest) to the server. But it works differently for editing - there's no guarantee you still have the file, and it wouldn't know where you saved it anyway. So, when editing, it loads the file from the server and puts it on something called an HTML Canvas. The bug comes from my misunderstanding of how canvases work - turns out, they have zero support for GIFs. So when you tell it "hey, gimme the base64 of whatever is loaded in that canvas right now", it takes only the current frame. It's so stupid and counterintuitive. Maybe I shouldn't have assumed it'd work in a way that made sense (it is Javascript, after all), and I'm still figuring out the best way to handle this in the rewrite.

And you know what? There's at least three more I should put here. I really want to. There's another dozen I could put here but aren't as interesting. But I've held my head in my hands three times already reading stuff, so I'll leave it there.

The point is - this has been one hell of a learning experience.

thellere -

Might fuck about and make a separate blog for the robot actually

thellere -

yes that’s right, connect to the open wifi access point you saucy little minx

fallow -

coffee said: #i thought this was robot thell for a sec #but this is 100% real thell

robot thell??? *amused/concerned combo*

babushka -

1) same

2) go on thell's blog and if you see posts tagged "robot testing" that means a bot wrote them

waterfall -

the reason it takes a second to post things on waterfall sometimes is because @thellere has to check each post we make and click/drag it to the right blog

thellere -

Confirming this is how it works

I am SO fucking called out by the outfit comment


thellere -

I just checked the database because I'm writing a dev blog and there's 169 variations of the NSFW tag

thellere -

Got logins/sessions working in the new frontend, that was expected to be one of the major bottlenecks

thellere -

In the mood to write a dev blog. Are y'all more interested in the tagging/search overhaul, or the challenges of the new UI?

justaharpy -

I feel like I have no where to go when it comes to finding an active blogging site. I feel uncomforable with waterfall rn because I heard that the creator is a racist??? As a black person I feel very uncomforable with that. pillowfort isnt very active rn and mastodon is more like twitter when it comes to layout. nothing is really filling the niche for me.

beefox -

iirc a while back thell (the creator) was a lot less open minded, but since has been working on it, as well as there are PoC on staff actively working to make waterfall a safe place

(a while back being 5-10 years ago i think? either way am remeber him talk about in the support discord before)

justaharpy -

https://babyfairy.tumblr.com/post/185282302372/actually-id-really-like-to-address-my-problems actually the person who he was being racist to didn't accept his apology. Tbh blaming on your racism for being in the uk and calling black people "blacks" is a shitty thing, I wouldn't accept it either.

thellere -

I'm sure like 90% of the code I've written today is a hack but fuck it, it works

thellere -

Has three best girls I wanna draw but doesn't know what order to do them in

If only I'd finished fuckin poll posts

thellere -



fandomsandsuch reblogged staff
staff -

Apologies for the unplanned downtime this morning! We had to do some maintenance to prepare for the API rollout, and it took significantly longer than expected.

This is the first of two backend patches to prep for the overhaul. The second one will be done at the time of the rollout - retrofitting the existing site to handle the data structure changes would be, in effect, wasted work.

There's still a fair bit to do before then, but in the meantime, you should notice some performance improvements. Thanks for your patience!

thellere -

So apparently those two contracts that were cancelled were cancelled because we explicitly offered to let protesters use the site

Fuck me for trying to be a good person I guess

thellere -

Building the new search system is gonna be a pain in the ass so I might just design it so it uses the current one for now and then just sorta slide in the new one post-release to save me some time

thellere -

I forgot to give them a shoutout in the last community update but @taxonomy-blog is one of my faves and deserves way more attention

thellere -

Was gonna reply to the shill but they blocked me because I made a tweet about how PF's invite system is artificial scarcity


Waterfall only has five staff members, official or otherwise, only one of whom actually codes or works full time. I'm not really sure what point they were trying to make by emphasising that Pillowfort "only" has 6 (and presumably more since that's only the "official" staff?)

sidh -

Wow it really is just that easy to summon that one pillowfort 'totally not affiliated just really into the platform' shill

thellere -

There's two or three that are REALLY prevalent. One of them even tried to advertise on one of my tweets, despite my bio literally saying "I made Waterfall"

thellere -

Oh lmao I just saw their assertion that we've "paid for extra services"

I can assure you we've not, I'm so averse to external stuff in Waterfall that I've literally reimplemented free public code

I literally wouldn't trust an outside agency to touch Waterfall, EVERYTHING here is done in house. Nobody external has been paid outside of lawyers.

thellere -

Tries to learn marketing to start getting WF out there but can't look at myself in the mirror for half hour after every time I try and come up with something for it

thellere -

Plugs the Patreon rq because the contract I was relying on to pay the staff just cancelled on me without notice

thellere -

Oh lmao apparently that one really prolific pillowfort shill I've been noticing on every tweet mentioning Waterfall is pillowfort staff, wording in some of their tweets gave it away, I just noticed

Whenever someone makes a tweet talking about Waterfall, there's one account that always hops on going "use Pillowfort, here's a free invite". I checked their tweets because I got suspicious and a) not only is there not a single tweet that's not PF related, they b) Explicitly say "thank you, the mroe people we get the better servers we get"

Which at least strongly implies it's a staff member's smurf account.

Maybe I should start being less picky about our advertising being less intrusive.

Community Update - June 2020

staff -

36.2215405525 squared

In May, we...

  • Upgraded the servers in preparation of the app

  • Made more headway on the new UI

Site Stats

As of writing, we have 30,105 users, 49,755 blogs, and 367,979 posts.

Site News

Most work right now is related to getting the new UI done. We're happy to report that we've gotten the first set of data from the site to display on there, signifying the first tangible progress, and frankly, giving a big morale boost. It's still ugly as sin, but real progress is being made now. 99% of code written so far is directly transferrable to the app versions too.

As a micellaneous site stat -the Commissions Market is now consistently handling an average of $100 of transaction volume a week. We're hoping that the reduced fees are helping those artists using it. We've also not had any bug reports for a while, which is excellent - we'll be keeping the Alpha label until we've figured out PayPal integration and written commissions though.

On a more serious note, things are a bit fucked at the minute. You can probably infer our stance if you used a calculator on the number at the beginning, but to be explicit on our stance - Waterfall is a safe space to discuss and organise. While we can't legally ignore any warrants or court orders, and so can't guarantee it's perfectly safe, we invite anyone who can make use of it to use Waterfall to post resources. If we DO recieve a warrant, we will fight it where we're able to do so. Our hope is that not being US-based makes it not worth their time to try and get us to comply.

Provided this line is present, Waterfall has not recieved or complied with any warrant in relation to the protests. Should this line not be present, assume we've been forced to hand information over.

While not in a position to protest directly ourselves, our lead dev is taking a break for a couple of days to work on software tools specifically to assist those protesting, but in the meantime, please feel free to use Waterfall for masterposts, or anything else - as a matter of policy, staff will not remove any posts, users, or blogs providing resources or organising protests.

If you are involved in protesting - please stay safe out there.

thellere -

palms sweaty, database spaghetti

thellere -

I know I've asked it before but I cannot for the life of me remember the answer

What features are we missing that PF has that are actually worth having here? Still mostly doing the solo dev thing so I need to triage what's important and what's not

thellere -


Might make this properly my personal so I can turn the no minors flag on and make the thelldev URL point to another blog where actual dev stuff goes, make things easier