None of this experiment is evident

None of this experiment is evident

A conversation on Irational server, Kate Rich and Femke Snelting

This conversation took place in two parts commencing on a terrace in Brussels in Summer 2023, a few months before starting the process of moving TITiPI’s email on to the Irational art server, away from Gandi, a domain name registrar that was recently bought up by Total Webhosting Solutions. At the time, Irational.org was in the process of transferring their web and email services onto a micro computer that would be hosted at Servus, an art network provider in Linz. Ten years earlier, we had discussed the server practices of Irational.org over pancakes at Femke’s kitchentable in the context of Are You Being Served?, the Feminist Server Summit organised by Constant. We returned to pick up the conversation in Spring 2024, once the move was done.

Femke Snelting (FS): To start, can you introduce the Irational.org collective, and how the Irational server came about?

Kate Rich (KR): Irational.org began in the 1990s, I joined in 2010. At that time Irational was more active as an art collective who were operating a shared server. Then in 2016, coinciding with a server upgrade, we disbanded and re-formed specifically as a Linux server collective, which is a better description of what Irational is now. A small group of artists who are interested in running a Linux server together. Currently it is four of us (Heath, Vahida, Inari and me) who split the operating costs – with around 8 additional server users or guests. I do most of the server administration – that side of things is unevenly distributed.

Part of Irational's interest has always been in running a physical server, not a virtual machine as is nearly ubiquitous now. When I started out as Irational sysadmin, we were using rescued servers from commercial server farms – fully working computers that were being thrown out as part of the perpetual cycle of upgrades. We would install Linux on a discarded machine, and get low-cost hosting at a server farm in London. Then when our previous server was reaching the end of its life, this was in 2016, we also felt a tectonic shift. There was a large-scale move to virtualisation, people weren't running individual physical servers in the same way as before, and the supply of deprecated rack mount servers seemed to have dried up. So for the first time we bought a replacement server brand new, and drove it to Cheltenham in west England where we installed it in a server farm run by gamers – one of the few places we could find that was still prepared to host a single machine.

FS: Then after seven years with the gamers, you started making a new move with the server?

KR: At the end of 2022, we got an email from Galaxy Web Solutions, our colocation hosts. Colo means we were effectively subletting rack space from them, at the gaming farm. It was the first year of the war in Ukraine and electricity prices had sky-rocketed. We'd been paying £50 a month for the Irational rack space up till then – this was going to increase to £150.

We had a long discussion in Irational about the options – including taking on more paying users (such as other art collectives), repurposing the server to heat a small scale agriculture grow room (a serious suggestion that never made it past the idea stage), relocating it to Serbia, where Vahida lives (electricity is cheaper there but the infrastructure is more unstable) or just accepting that the whole prospect of a physical server had become untenable. The other route was to figure out how to make the server much more energy efficient. Part of our operating philosophy for the Irational server is to do as little as possible – keep it low-maintenance and not get caught up in the busy work of upgrades, improvements and obsessive security work. That means not running any resource-heavy content management systems like Wordpress or even Etherpad. So the CPU load on our server is actually quite low.

FS: It seems quite excessive the gamers asked a hundred and fifty pounds if you're not using that many resources?

KR: The amount that Galaxy was paying for their rack had just quadrupled, due to the electricity price shooting up. And it's a flat rate - I guess calculated on the idea you would be running your hardware as hot as possible, to maximise output – not Irational's operating method at all. But we were also aware that our server – a Dell PowerEdge R320, this big chunk of metal with an always-on fan – was still drawing a lot of power at least 40 W, even though it was mainly idling. So we were generating a lot of hot air on with this old-school rack server arrangement whose time is fundamentally gone. The discussion then turned to – can we work within the limits of something small and light instead - the kind of hardware that didn't exist when Irational began? That was the start of the idea to move all Irational web and mail services onto a solid-state microcomputer. Inari did some research and we settled on the Raspberry Pi 4. Then following a meeting in Rotterdam between Inari and Davide from Servus in Linz, Davide offered to host the Pi at Servus, for a trial period monthly fee of only €5 – based on an estimated maximum power usage of 5.85 kWh per month. It took us over a year, working at a snail's pace, for all that to actually happen.

FS: What happened to the old Irational server in the meantime?

KR: We kept it running for a full year while this was happening. Paying a hundred and fifty per month and running down the Irational bank account. In some ways it looks very decadent, burning £150 a month on physical hosting when we could have immediately taken the virtual route, spending £2/month or whatever, to be exported to some ghostly location (or locations). But in another way it's what the collective funds are for – the space to experiment. It did take forever – all of us moving very slowly, just due to the interventions of life. But the lack of efficiency in the collective is something that I actually treasure.

For the first step, Inari offered to research the hardware components. He is in Rotterdam, so he bought them online there and mailed them to me.

FS: You were in Bristol?

KR: Yes – the model of Pi we wanted was only available in Europe at the time, and the vendors wouldn't ship to the UK, which has become increasingly common since Brexit, with every cross-border transaction taking on a lot of extra friction. So Inari shipped the Pi to me, along with a hard case, for dust protection. I bought a 500GB M2 SSD and a power supply, then I sent the whole thing to Heath in Devon to format and partition the disk which takes more understanding of hardware than I confidently have, and to install the basic operating system. When that was done, Heath brought the Pi back to Bristol and we set it up in my flat to do the rest of the install. But it is difficult to actually configure a server from a home broadband connection. So after installing the generic software packages we mailed it to Servus in Austria and did the rest of the configuration remotely.

FS: You mean it is difficult to do that work from home because every time the IP address changes, you have to reconfigure?

KR: Yes. To set up a valid web or mail domain you need a certification authority to verify that you are the real instance of that domain and not an impostor. So in order to configure anything, you need to point your domain names to a verifiable IP address, which was not possible while the Pi was on my home broadband. There is also the security side – we were mailing the server to Austria, so any data on it would be potentially insecure while in transit.

FS: So Davide at Servus offered to plug the Raspberry Pi into the Servus network and that would mean it would have a stable IP, network and electricity.

KR: Yes. That took a long time also.

FS: We started this conversation when I saw you make a note in your diary saying “Where is server?”

KR: That was when it was lost in the mail! I sent it via Royal Mail tracked post on Friday August 13th and about a week later I suddenly realised it hadn't arrived.

FS: From where to where?

KR: From Bristol to Linz. Because of the Brexit situation I first thought, should I wait and bring it with me on Eurostar when I come to Brussels to talk to you, and post it from there? But my bags were already overloaded with 5L olive oil and two bottles of wine. So I did some research at the post office in Bristol. If a parcel is over a hundred and thirty five Euros in value, the receiver pays VAT and duty on it. If it's over thirty nine Euros and a gift, they still pay VAT. You actually can't mail something from A to B without it being a transaction, a change of ownership – your only choices are gift, sale or commercial sample which is bizarre, because you do this all the time in life – move things around without it being a transaction – but in the postal system there is no category for sending something to yourself.

The concern wasn't so much the cost of duty or VAT – even thought it was absurd, having just purchased the Pi from Europe to pay all the taxes again to send the same thing back to ourselves there – but the level of admin it would add for Servus. So I ended up putting the insured value as €120 ( in case it got lost in the post, which also seems to happen increasingly often) and then scribbled on the customs label to make it look more like €20. Perhaps this is the reason it was stuck at the border? Anyway, when we pulled up the online tracking, we could see it was being held at Customs in Austria – this went on for 11 days, then it was released for delivery without explanation.

Things continued to move slowly. When it arrived at Servus, Davide was away, so it waited around again. Then there was a stretch of confusion with the Servus technical team about assigning an IP address, where we were mutually not comprehending what we were being asked to do, which is typical of technical conversations. Then finally the server had a permanent IP address on Servus' network and we could log in remotely and do the config – I think that took another three weeks. It was not about being ambitious – I just wanted to duplicate the setup we had on the previous Irational server, doing something more interesting with it could come later. But you are moving from Debian 9 to Debian 11, and the different architecture of the Pi, and a lot of the software packages have also changed since 2016. In addition there are the escalating security demands of the big platforms – in order to be able to send mail to a gmail address, you now need to do comply with a whole pack of security protocols, DKIM , DMARC, SPF, which we had been getting away with ignoring up to that point. Getting mail right was the killer. We could all survive with the web server going down for a while but email – that's life or death. And I only have a certain amount of daily headspace for this kind of thing.

FS: Do you think the Irational server as an art server project is about that tension, exactly because it hosts services you really depend on?

KR: I don't think any of us have significant other mail accounts – it's not like we are making a show of using Irational mail but are really using a university email account or gmail in the background. So when email goes down it's critical.

FS: So that means there's something at stake.

KR: My whole experience with sysadmin is being invisible but the minute something goes wrong, everyone is in contact, with urgency. With mail in particular, it seeps into every aspect of life – being without mail even for a few hours can feel like being locked out of your home and work and social and emotional life simultaneously. It's quite scary to administrate. Making everything more difficult, the DNS tool we use to manage the domain names, which is provided by an old friend of Irational, is now only accessible through a text-based interface. This meant typing the configuration into a blank text document, rather than a nice graphic user interface where you can choose options from a drop-down list. It's like working in the dark.

All of this is obviously not about efficiency in terms of either time or money. But there is a real resistance to being Cloud-ready.

FS: Why this resistance to virtualization – and what does it mean to be Cloud-ready?

KR: I find it difficult to describe. I guess I would first ask the opposite: what's this immediate acceptance of virtualization? It's a huge infrastructural shift that everyone seems so happy to jump into without question. Should we just be careful about this thing and think first about what is it and what does it mean?

FS: Of course, virtualisation is hardware emulation, it literally immaterialises the material. So it means that also the relation to the server as an object becomes virtualized, which if that's a source of pleasure, it is the relation you don't want to erase.

KR: It's not necessarily pleasure! It's more like a source of knowing and taking the opportunity to be present. In other realms of existence – it's completely accepted that might you want to build your own boat, or be a beekeeper, where those things are clearly also not for everyone. Many people would buy a boat. But if you want to build one that is not contentious, even when it can likely be an endless source of frustration and a financial drain and a huge amount of labour – it's widely acceptable and admired. Or growing your own vegetables – there are pests, adverse weather – but allotments are very mainstream. Which makes me think, where are the allotments for the physical server operators? You would have your shed and you come out occasionally and chat with the the other people – Ah geez, so that's how you've got mail configured? That would be amazing.

FS: The allotment is helpful to think with, even if I have no gardening practice to speak of! It brings up the image of multiple non-virtual servers existing next to each other. People that take care of servers would find themselves on the same area of land but each in their own corners or on their own plots, where they can still sort of look over each other's hedges, to see how things are growing next door. It goes against the culture of secrecy and security you were alluding to with mail. It's also a nice counter image to the virtualized server, where you would supposedly rent an autonomous space on a server but in reality, this “space” is distributed over many machines, depending on how the service provider decides to allocate resources. And while you are technically sharing space, you're socially and politically alone on a virtual server – the point of virtualization is that the different plot holders are made unaware of each other.

KR: The physical conditions appear to vanish. Virtualization also enables the plantation model of infinite scaling and replication – that adding more users could be effortless and exponential.

FS: I think that's where for me, a more basic resistance to optimisation aligns with resisting Cloud-readiness. Virtualization does this trick of infinite expansion, promising flexibility and efficiency but n the end it is always about scaling up. It exports this trick into every aspect of life – I find that really scary. The technology of virtualization co-produces what we at TITiPI call “The Cloud regime”. It impacts the way we can deal with technological knowledge, how we might be able to maintain infrastructures together, because we have come to expect these infrastructures to be always available. Virtualization makes all that go away, and the work involved or the resources needed to keep it all going become unimaginable or unspeakable.

KR: It's getting into the infrastructure that is the important part to me. Not only the computer hardware, but plugging that in at this location, considering the energy consumption, whose roof it's under, how it travels there, and all the relationships and negotiations around that. For me, running a physical server is a stand-in for going deep into the infrastructure in general. Just in this one element, it's not being DIY everything, I'm more than happy for someone else to fix my bike.

FS: Right. I enjoy your work on the Irational server a lot, but I'm also really glad to not have to do this myself. Is this division of labor still a form of efficiency? Wanting someone else to repair your bike could still be a form of optimization, of skills and expertise.

KR: It could be but that's not the reason for me. It's not as if the time I save not learning how to fix the bike is invested to enhance my sysadmin skills - it's that I'm genuinely not interested. My sysadmin style is anyway super-amateur, by design mainly doing internet search and not panicking when things break. But I also rarely use optimizing as a standard for anything, and actually I think that applies more widely. There's a certain mindset that is very concerned with optimisation, but not everyone has that, or they do but only around certain things. This idea that everyone wants everything to be efficient all the time – it varies hugely across people and contexts.

FS: How does this all play out for a service like mail specifically? Mail is so much part of professional and social life. It takes a bit of effort to experiment with it, to assume it might not be available all the time. It means to accept that when something breaks, fixes cannot be immediate because it takes actual people to find a solution and they might be occupied elsewhere. So, okay, well, maybe TITiPI won't have any mail today and we just deal with that.

KR: Yes, well, if mail breaks it is not that relaxing for me, I have to drop everything. But it's a one-off intervention –you go into survival mode and you fix it. Just to get it stable again, you don't try and improve it. You definitely don't try and make it sleeker because that's when you're going to break something. You just want stability with mail, for it to be there. The main internet mail protocols IMAP, SMTP, POP3 have been around since the 1980s and are built with that in mind.

FS: So would you say that it's a resistance to optimisation? What's the Irational approach?

KR: Maybe it's a survival or stability approach that is almost counter-improvement. It's also counter-growth - we're trying to keep the server small and stable, that's the value.

FS: When we last talked about Irational server, in 2013, you said something that I would like to come back to: “What I hate about technology: one person says, I have an idea, I can make it better and everyone has to follow.” This pressure to expand, how does Irational resist it?

KR: One aspect of the server that could be illustrative is how we've chosen to configure the mail server. The standard thing is to have virtual mail users. You're hosting a number of domains on your server and each domain has its own email accounts. Most people would install a virtual user database, then each domain can manage their own email addresses independently.

We've taken the very rare alternative to configure mail with system users. So if you want an info@ address for TITiPI, I would need to make an 'info' user on the server with its own Linux user account. It also means that any mail to info would go to that user, regardless of the domain – so info@irational.org, info@titipi.org, info@feraltrade.org – another domain hosted on Irational – all end up in the same mailbox. That's more than scale-neutral, it's scale-negative. Normally you'd be thinking ahead - another domain is going to join the server they're going to want their own info@ address. Or maybe a new domain wants to join, with its own kate or femke but those usernames are already taken, so we'd need to negotiate. This is perhaps counter-efficicency in action.

FS: It's a built-in antidote to growth.

KR: We're not going to end up with 100 domains - you would run out of options for email addresses, or you would be putting strange demands on users, which we did with TITiPI, to not use a generic email address like info@.

FS: This is how we ended up with titipi@titipi.org which is very nice. But the other thing about not having virtual users is that there's some kind of continuous awareness of other people being on the same territory, and that they could be in each other’s way. I think that's really important. Which disappears when you have virtual users, you're all completely segregated, or at least in theory.

Recently Constant decided to bring everything they had online in various places together and build a virtualized environment for the different services they are offering. One of the arguments was that because everything had ended up in different places, this produced lots of work, because each environment has its own requirements, logins, update speeds etc. It had become really unmanageable. This is of course relates to scale, when there are too many different things going on it becomes also really hard to hold.

KR: With Irational we limit how much can go on, and I'm fine with that. More can go on, it just can just go on elsewhere. The server operation has its own inbuilt limits, it's part of keeping things low-maintenance. For the user it's almost an encouragement to extend your work into other locations . I use Constant Etherpad and FoAM server for different services – it's like having keys to other buildings that you can casually visit.

FS: At TITiPI we’ve been looking into how the Cloud is changing the way that collective and institutional life is structured. How infrastructures co-produce institutions. So we've been thinking about how to do promiscuous hosting – the mission we gave ourselves was to set up a “non-sovereign infrastructure”, instead of making our own, independent server with everything on it, we try to experiment with different kinds of infrastructural interdependence. Some digital services we host ourselves, some are with Varia, mail is on Irational server, other parts of our infrastructure are cared for by Constant. And now we're working on hosting another befriended organization on our virtual web server, which is super nice.

This attempt to practice “non-sovereign infrastructure” brings a particular kind of digital discomfort that I am interested in. Trying to think with the seemingly inevitable presence of computational infrastructure in our lives, it seems that one way to not succumb to them as they are, is to resist their normalcy and performative comfort. Discomfort might be an entry to transform our relationship to Cloud infrastructures, to chip at their edges, without the illusion that we are speaking about alternatives.

Digital discomfort is a bit of a different approach then working towards DIY technological sovereignty, it seems like something that is much more open-ended, ambiguous and non-heroic. It seems to be more about working with different infrastructures that weave various regimes and modes together.

Going back to the conversation we had at Are You Being Served in 2013, the energy around servers seems so different now. Maybe it's also that my own perspective on it has changed. You were for example speaking about system administration on Irational server and you made a reference to it feeling military. Something that maybe isn't in there now in the same way?

KR: Some of the difference could be down to the context for that conversation, which was an event for and around collective servers. I found it both really exciting and disorienting to be invited to this feminist server gathering – feminist not being my experience of system administration on Irational at all! A security culture of server administration still dominates, it's built in to the software. And we still encounter a storm of automated hacking attempts from the internet every day, like passing through a perpetual meteor shower of ill intent, even on a tiny server like ours. It's almost the containing condition for the whole practice.

FS: OK, yes – an ongoing battle. But what I am trying to get at is that in 2013, we were already seeing that many server projects were kind of phasing out. You say that there's not so many other self-run projects any more. So this sense of server management as a thing you do has become something quite rare and special. I am wondering how much Irational’s insistence on running your own, non-virtual infrastructure is still motivated by an idea of independence, gaining autonomy, and some idea of security, even when you know that dropping data into a Google Drive might be less risky. How does that work for you?

KR: For some of Irational it might be or have been about those things – personally I'm sceptical that autonomy is even possible as a concept. I think we've all maybe got quite different motivations for running a physical server that we don't even feel the need to discuss much, or have consensus on, we just all want the server to be there, which is enough.

FS: Right, so even with different interests or things on stake for each of you.

KR: Especially with that – it's a stronger prospect, that we would all have different interests in it. What is common is having the server as something to gather around, like a campfire, or a bin fire on the street – occasionally we come together and warm our hands around it. Which is much less about autonomy than community. We would all still like the server to be more sociable with other server and users, but as you mention, there is hardly anyone left in the self-organised server community to socialize with – or people who are interested in servers in any way. For example just about every artist you meet has a gmail address, and is totally unbothered by that. It's not just about running servers – it's this awareness of a material relationship to computing that is generally missing. People don't get why you just wouldn't just virtualize. Why are we paying for something that you could have almost for free – it would still look the same! Hardly anyone finds the distinction worth considering.

FS: That's what is interesting to me here, the affordances of not making it function like how other things are run, or not do it the same way, even if it seems to look the same way on the outside.

KR: Running a mail server reminds me of the time I watched you improvise making bread rolls at home, from scratch. It took you about two hours of deep concentration, and we ended up with something almost indistinguishable from those ubiquitous white supermarket bread rolls. Amazing. A level of the mail server operation that I like is its unannounced nature. When someone receives an email from an Irational mail user, none of this experiment is evident. Or if it was, it'd be a problem – it's only when it's not working that there's any noticeable difference.

FS: I am interested in paying attention to processes that are usually backgrounded, either because they are not producing spectular results or generally too hard to pay attention to. When it comes to bread baking, I am less interested in the sexyness of wild yeasts, but rather enjoy what you’ve called the “miraculous normalcy” of white bread rolls; I never thought about this as a training for infrastructural sensitivity, but maybe it is. The impact of the Cloud regime often plays out in these unassumed corners of life. Every time I start up Thunderbird, I try to enjoy the quick message, 'Connected to Irational', a microsecond of presence that is super important. If it was a flashy new communications service – maybe it would be less exciting to me than the fact that it's email, which has become such a given as a free Big Tech offer, there's not even an imagination that this could be run by friends. I think the normalcy of the Irational webserver is indeed miraculous.

KR: I would probably give up the web hosting side of server administration without a struggle if we had to, and use other servers for that. It's mail that I don't want to give up, because operating mail is such a trick. It's hard to configure, but it has so much weight to it.

FS: When we were setting up TITiPI mail on Irational, we got glimpses into this sort of sedimented large scale infrastructures that you don't normally have to get your head around. Like configuring the TXT records to the domain name zone, which authorise your sever to send mail from that domain, without these you can no longer send mails to Gmail, as you mention earlier. I don't have the same stamina for it as you do, to maintain this kind of thing on a continuous basis, but it was an amazing adventure for me.

Our interest with TITiPI is in the crossing between computational infrastructure and instituting otherwise - and it seems like mail is such a good carrier for that interest. And of course it's also the address of the institute, through the emails we send out. Even when it's barely or not at all visible to most people.

It feels like the rise of the Cloud and virtualization has overtaken many interesting server practices or … sandboxed them. “Sandboxing” is a practice in software development where you zone off experiments into isolated containers or sometimes virtual servers. The virtual server acts as a stand-in for a whole world without actually reaching out to it. This is the Cloud Regime. From a security perspective it seems of course good practice to not experiment on a live service, but as the Cloud enters into all areas of life, sandboxing seems to become an approach to anything and everything which I find really frustrating. This to me feels very precious in the Irational rationale, the persistent interest in not doing as-if, and I figure that this has a lot to do with your refusal to be Cloud-ready. When something goes wrong with a virtual server, the imaginary is that because it exists within an independent container, you can just start again and reinstall without consequences. Of course that is never really the case, but I think my point is – because there is less at stake, the type of experimentation that can happen on virtual servers is limited, it’s risk calculation, not life. But for Irational, it seems it's about transforming that which is at stake, an experiment while it's happening. The live wire.

Brussels, August 2023 - April 2024